Grey Panther's Place

An interesting proof for Pythagoras’s theorem

Attila-Mihaly Balazs — Thu, 05 Jan 2017 19:12:00 +0200

I recently saw an interesting proof for Pythagoras’s theorem in the MathHistory series which I wanted to share with y’all :-)

So a quick reminder, Pythagoras’s theorem says that if we have a right-angle (90 degree) triangle, then there is the following relation between the length of the sides:

a = sqrt(b^2 + c^2) (where a is the length of the longest side) - and vice-versa.

The proof goes like this: lets rewrite the formula like a^2 = b^2 + c^2. We can interpret this geometrically as: (for a right-angled triangle) the are of the square constructed on the longer side is equal to the sum of the areas of the two squares constructed on the shorter sides.

And now the proof goes as follows:

consider a right angled triangle

“clone” it 4 times and put it together such that the longer sides form a square. Now the area of the inner square is a^2 while the area of the big square is a^2 + 4*At (At is the area of a triangle)

rearrange the triangles as shown. The outer square is still of the same size (the length of its side - a+b is the same) but now it can be written as b^2 + c^2 + 4At. Hence a^2 + 4*At = b^2 + c^2 + 4At which can be simplified to a^2 = b^2 + c^2, or if you prefer to a = sqrt(b^2 + c^2).

I only had one nagging feeling after seeing this proof - how do we know that the first big square constructed is actually a square. Can’t it be that its “edges” are not lines, but slightly crooked like below?

Fortunately we can use the fact that the angles in a triangle add up to 180 degrees (ie. a straight line) and show that the sides of the external triangle are indeed straight lines:

Finding the N-th word in a complete dictionary

Attila-Mihaly Balazs — Mon, 02 Jan 2017 13:28:00 +0200

Problem statement

Find the N-th word in a dictionary which contains all the words that can be generated from a given alphabet of length at most M (and sorted by the conventional dictionary sorting rule / lexicographical order).

As a short detour: why did I become interested in it? It was during my investigation of the upper limit for the number of strings formed from a given alphabet that can be encoded in a given number of bits. Even more concretely: what is the upper limit for the length of a DNA/RNA string formed from nucleotides (ie. a string with alphabet [A,C,G,T]) that can be encoded on 64 bits. Note: the problem statement that we need a codec (ie. both enCOding and DECoding, so we’ll solve a bit more generic problem than just the one-way one described in the title).

The first solution which came to mind was to use some bits for the length and the remaining bits to encode the nucleotides (2 bit / nucleotide) however the question remained: how many bits for the length? And is the solution optimal?

So finally I came up with the following formulation: consider that we have a dictionary of all the possible nucleotide strings for length at most M. Now let the 64 bit value just be an index in this dictionary. This is guaranteed to be the optimal solution (if we assume that the probability of occurrence for every string is the same). Now we need three things:

what is the largest value of M for which the index can be stored on 64 bits?
a time and space efficient way (ie. not generating the entire dictionary and keeping it in memory for lookup) to get the index of a given string (the enCOde step)
the same to get the word at a given index (the DECode step)

There is also a somewhat related problem on Project Euler (24: Lexicographic permutations) - that wasn’t the inspiration though, I found out about it later.

Some initial observations

Just by writing out the complete set of words of length at most M formed from a given alphabet we can make some observations. For example consider the alphabet [A,B] and write out:

the words of length 0: '' (the empty string)
the words of length 1: A and B
the words of length 2: AA, AB, BA and BB

So pretty quickly we can see that for a given alphabet and a given length we have exactly len(alphabet) ** length possible words (where ** is the exponentiation operator - ie. a ** b is the b-th power of a), since: we have length positions, at each position we can have one of the len(alphabet) characters, thus the total possibilities are len(alphabet) * len(alphabet) * ... length times which is len(alphabet) to power length.

After this we can ask “how many strings of length less than or equal to M are there”? (question 1 from the initial problem statement). This is simply sum(len(alphabet) ** i for i in [0, M]), also known as the geometric progression: (1 - La ** M) / 1 - La where La = len(alphabet).

So for example if we have the alphabet [A, C, G, T] and 64 bits available we can encode at most 32 characters according to Wolfram Alpha.

Finding the index of a string

To find this we just need to count how many strings there are in the dictionary before our string (remember the dictionary is in lexicographical order).

A concrete example: our dictionary contains all the words of length at most 3 (M=3) formed from the alphabet [A, B]. What is the index of the word BA? (we consider that index 0 is '' - the empty string, index 1 is A, index 2 is AA and so on).

What is the position of BA in our dictionary?

If we would only have words of length exactly K we could compute this by considering BA a number in base 2 (binary) where A=0 and B=1, transform it to base 10 and have our answer (ie BA -> 10b -> 2 -> BA is at position 2 - or is the 3rd word - in the dictionary AA, AB, BA, BB).

However our dictionary contains all words of length exactly 0, 1, 2 and 3. So just consider each in turn!

In a dictionary containing the words from the alphabet [A, B] of exactly length:

K=0: BA would have index 1
K=1: BA would have index 2 which is the same as indexOf(B) + 1
K=2: BA would have index 2
K=3: BA would have index 10, which is the same as indexOf(BAA)

So, to find the index of a string:

Go from 0 to M (the maximum length allowed for words in our dictionary)
Generate a word of length K from our word by either (assuming our strings are zero indexed):
Taking the characters 0 to K (exclusive) if K < len(word)
Padding the word with the first character of the alphabet up to length K
Finding the index of this (sub)word in a dictionary that contains words of length exactly K by considering the (sub)word as a value written in base La (La == length(alphabet)). Add 1 if we’re in the first case since the longer word would come after the shorter ones.
Sum up all the values

Or in Python 3 code:

def indexOf(self, word):
    assert len(word) <= self.__max_len
    result = 0
    for i in range(0, self.__max_len + 1):
        if i < len(word):
            subword = word[:i]
            result += self.__valueInBaseN(subword) + 1
        else:
            subword = word + (i - len(word)) * self.__alphabet[0]
            result += self.__valueInBaseN(subword)
    return result

Finding the N-th string

Finally getting at the problem stated in the title. For this I noted how the dictionary can be constructed for length M:

the dictionary for M=0 is just '' (the empty string) and for M=1 the empty string plus the alphabet itself.
for M>1 take the dictionary for M-1 and prefix it with each of the characters from the alphabet. Finally add the empty string as the first element.

So for example if we have [A, B] as the alphabet then:

the dictionary for M=1 is 0: '', 1: A, 2: B
to construct the dictionary for M=2 we replicate the above dictionary 2 times, first prefixing it with A, then with B and finally we add the empty string in front:

0: ''  1: A    4: B
       2: AA   5: BA
       3: AB   6: BB

This suggests an algorithm for finding the solution:

take the value. Decide in “column” it would be.
you know the number of words in each column: len(dictionary) - 1 / len(alphabet)
len(dictionary) is sum(len(alphabet) ** i for i in [0, K]) (see the initial observations)
this can also be precomputed for efficiency
the column index gives you letter index in the alphabet
now subtract from the value the index of the first word in the given column. If you get 0, stop.
Otherwise make K one less and look up the new value in the dictionary of length at most K.

A small worked example:

lets say we have [A, B] as the alphabet and M=2. We want to find the word at 5 (which is BA if you take a peak at the table above). So:
in each column we have 3 words, so 5 is in the 2nd row (the row with index 1) which gives us “B” as the first letter
now subtract 4 (the index of the first word in the 2nd column - B) from 5 which leaves us with 1
now find the word with index 1 in a dictionary with M=1 which is “A”
thus the final word is “BA”

Or in Python 3 code:

def wordAt(self, index):
    assert 0 <= index <= self.__lastIndex
    result, current_len = '', self.__max_len
    while index > 0:
        words_per_column = self.__wordsPerLetterForLen[current_len]
        column_idx = (index - 1) // words_per_column
        result += self.__alphabet[column_idx]
        index_of_first_word_in_col = 1 + column_idx * words_per_column
        index -= index_of_first_word_in_col
        current_len -= 1
    return result

Note: you can find a different algorithm to do the same on math.stackexchange.com, however I found the above to be visually more intuitive.

Can we do it simpler?

So we solved the initial problem (both the one stated in the title and the one which motivated this journey) however it took over a thousand words to describe and justify it. Can we do simpler? Turns out yes! We just need abandon our attachment to the lexicographical order and say that as long as we have a bijective encoding and decoding function with the property decode(encode(word)) == word we are satisfied.

A simple and efficient function is the transformation of the word from base La (length of alphabet) to base 10 and vice-versa. For example if we have [A, C, G, T] as the alphabet and GAT as the word we can do:

encode: 2*(4**2) + 0*(4**1) + 3*(4**0) which is 33
decode: 33 is written as powers of 4 as above and 2, 0, 3 corresponds to GAT

Again, the ordering will not be lexicographical (A, AA, AB, ...) but rather a numerical-order kind-of (A, B, AA, AB, ...) but the algorithm is much simpler and in the case that La is a power of two, very efficient to implement on current CPUs since division / remainder can be done using bit-shifts / masking.

More speculation

I didn’t actually want to encode DNA/RNA sequences, but rather mutations/variations which are pair of sequences (something like G -> TC or GT -> ''). Now I could just divide the 64 bits into two 32 bit chunks but the same initial question would arise: is this the most optimal way for encoding?

So we go the same solution: what if we would have a dictionary of all the variants ('' -> A, '' -> AA, ...) and just index into it. How would we construct such a dictionary and how would we order it?

Turns out there is an algorithm inspired by the proof that there are the same number of natural numbers as there are rational ones. However that doesn’t give us a way to find the N-th element in the sequence but a Calkin–Wilf sequence does.

So we can have the following algorithm:

represent the pair to -> from as two numbers A and B (refer to the discussion until now how we can do that)
use the Calkin-Wilf sequence (combined with the continued fraction formula) to find the index of A/B
or conversely use the sequence to transform the index into the A/B fraction and then transform the numerator and denominator into the original sequences

This is just speculation but it should work in theory. Also, it is fairly complicated so perhaps there is a better way to do it by making some simplifying assumptions? (like us eliminating the lexicographic ordering requirement).

Source code

A complete implementation of the above algorithms (with tests!) in Python 3 can be found on GitHub.

The limits of science

Attila-Mihaly Balazs — Thu, 08 Sep 2016 10:44:00 +0300

In a lot of ways Science has become the religion of the day. We can’t go more than a day to hear / see / read a “news” story about “scientists” saying something about something important. We can’t help but feel dazzled, confused, perplexed, overwhelmed by these announcements. And we have discussions with others: - I heard that scientists say that X cures cancer. - Didn’t you hear? Scientist announced that X causes cancer How is this really different from saying “my shaman said”?

The core principles of science

I’m going to argue that scientific thinking is really in a different league - but it also has many limitations. The core ideas of scientific thinking are:

We can observe things and those observations have some relation to the objective reality
This objective reality is probabilistically deterministic (ie. if we flip a coin it will land either heads or tails but it generally won’t turn into a unicorn and fly away)
Based on our observations we can construct models / hypothesis about cause and effect. While there can be an infinite amount of hypothesis, we assume that reality has a simple and elegant structure and thus prefer the hypothesis which explains the largest category of events with the minimal assumptions
Scientific hypothesis need to be testable (also called “falsifiable”) - ie. they need to be forward looking / have predictive power. For example if I say “object alway fall downwards” then you or I can take a ball, a key, a rock, etc and verify that indeed, when I let go of it, it falls towards the ground. However such positive outcomes have little value (they increase the likelihood that the hypothesis / theory is true only by a small amount) and the most valuable things are falsifications - when somebody makes a prediction based on the theory but then a different outcome is observed - which most likely means that the theory is false (unless there was an error in the experiment).

This looks an awful lot like “the ten commandments of the science religion”, doesn’t it? Because there is no reason to believe them intrinsically. There are only two things in favor of this mindset:

This is how our world seems to work - and thus these principles seem intrinsically true to a lot of us. Then again this “gut feeling” is no different from being convinced that a given kind of deity governs our lives

What is different from every other religion/philosophy however is that it contains the framework to extend it. You just observe, theorize and then try to falsify your theory. Voilà! You’re adding to the scientific knowledge. All other religions are closed while the religion of science is open.

Science is not “truth”

You might have observed that the principles - as described in the previous section - have somewhat of a wishy-washy nature. I keep using words like “likely” and “usually”.

For example I said “if we flip a coin it will land either heads or tails but it generally won’t turn into a unicorn and fly away”. I can’t say for certain that it won’t turn into a unicorn, but so far nobody reported a case of this happening, but we observed the coin landing heads or tails a lot of times so we’ll assign a very small probability to something else happening.

Scientific results are always probabilistic, but that is just life: if I go out tomorrow I might get hit by a car, but I most probably won’t. Note that other religions generally avoid saying anything about “this life/world” and reserve their proclamations for “the other/after life”. At least science is trying to make some predictions, even though they sometimes turn out to be false.

This bears repeating: Science is not “truth”. There is no such thing as a “scientific fact”. Our scientific knowledge is simply a collection of theories which we failed to falsify - as of yet. Now, some of those theories have been around for a long time and have been found true in many situations and it’s prudent to do to act as if they were the absolute truth, but we must accept the fact that there is always the small possibility that they might turn out to be false.

To pound some more this idea: all the above is true even if we would “do science perfectly”. However we are humans: subject to our biases, feelings and other motivations.

Science is not math

(a couple of words about induction)

Related to “there is no scientific truth” is the mirage of mathematics in science, which goes something like this: “we all know that 2+2 is 4, this scientist is using a mathematical formula, so whatever comes out of the formula must be true”.

No!

Mathematics and science use very different ways of reasoning:

In math we use deduction: we state some ground rules (“axioms”) and from those we deduce (prove) all the other statements. All the deduced statements are “true” (if we didn’t do any mistakes) since they are derived from the axioms which we assume to be true. This doesn’t mean however that it necessarily has any relation to the objective reality. (And just a side-note: even if we stay within the bounds of such imaginary systems - ie. don’t try to apply it to the “real world” - we will hit some fundamental limitations ¹)

Science however uses induction: if I see that both an apple and a rock is failing down, I guess that both “falling down” events happen because of the same cause. Such “rules of thumbs” ² work generally but can give the “wrong” results sometimes (as opposed to the rules of deduction which are always correct if we accept the initial axioms). Of course for a scientist these cases of “unexpected outcome” are the most interesting ones since they signal an opportunity to learn something new, but they can be most distressing if we think of science as “the source of truth”.

So how does science use mathematics? It is just a precise way to describe hypothesis / theories and to manipulate those. Thus we might say that the maximum distance of a ball thrown is described by the equation v^2 / g and we can make predictions about the distance of a ball will travel before throwing it (and verify after the fact that prediction was - mostly - correct) but the fact that we formulated the theory in mathematical term does not make it fundamentally more true than any other scientific theory. It still is “the best theory we have for now which isn’t refuted by evidence”.

Other limitations of science

Coming back to the idea “science is not the absolute truth even if we would “do science perfectly”:

We don’t do science perfectly:

We have biases and when we have a theory we might not look “hard enough” for evidence to refute it
The academic structure is not set up to encourage verification of results: publishing replication results or negative results is discouraged (may scientific journals don’t even accept them for publication)
Science is entirely probabilistic (it only tells you what probably is true), however statistics (the branch of mathematics which deals with probabilities) is complicated and it’s easy to make mistakes when judging what constitutes probably true³
In some very interesting fields it is hard to conduct experiments (more on this in the next section)

And as if the above wasn’t enough, there is the problem of communicating the results (the step where “there is a 60% probability that eating a bit of chocolate improves bone density in white women over 50” turns into “Science Fact! Women should eat chocolate daily!”).

To give some uplifting news: these issues are known and people are trying to work on it. There have been scientific journals set up to publish replicating studies and/or studies with negative results. There is a movement to encourage researchers to pre-register their studies (to state what data they’ll collect and how they will analyze it) and publish the results even if they are negative. Finally some journals require scientists to give a “simplified abstract” when publishing the research which can be adapted by journalists easier.

However these are hard issues and we can help out by not having unrealistic expectations.

Words about human-centric fields

A couple of final words about the science in fields like health (mental and physical) or economics: these are the fields which are the most important to us but where the difficulties presented multiply:

Generally we can’t do random double-bind trials where we select a group of people and infect them with AIDS or make them live on less than $1 a day (you know, because we value human life)
This means that we can only study people who already are in this situation but that makes it very likely that we confuse cause and effect
Even if the experiment is non-intrusive (or we’re doing an observational study) it is very hard to get a diverse set of participants (ie. most psychology experiments are done on young white males in the US - no wonder that they don’t replicate across the world)

Again, people are working on addressing these issues, but it is just one more reason not to believe the “fad” articles published daily and shared virally on social media.

Science is not perfect, but it’s the best that we have.

https://en.wikipedia.org/wiki/G%C3%B6del%27s_incompleteness_theorems ↩
https://en.wikipedia.org/wiki/Mill%27s_Methods ↩
Is Most Published Research Wrong? - https://www.youtube.com/watch?v=42QuXLucH3Q ↩

A fresh start with Pelican

Attila-Mihaly Balazs — Mon, 04 Jan 2016 21:18:00 +0200

Here we are in 2016, trying to start blogging again. Using Pelican is more complicate than it needs to be :-(.

On benchmarks

Attila-Mihaly Balazs — Fri, 28 Mar 2014 09:17:00 +0200

Numbers every programmer should know and their impact on benchmarks

Disclaimer: I don’t mean to be picking on the particular organizations / projects / people who I’ll mention below. They are just examples of a larger trend I observed.

Sometimes (most of the times?) we forget just how powerful the machines in our pockets / bags / desks are and accept the inefficiencies of the software running on them. When we start to celebrate those inefficiencies, a line has to be drawn though. Two examples:

In 2013 Twitter claimed a record Tweets Per Second (TPS - cute :-)) of \~143k. Lets round that up to 150k and do some back-of-the envelope calculations:

Communication between the clients and Twitter: a tweet is 140 bytes (240 if we allow for unicode). Lets multiple the 150k number by 10 (just to be generous - remember that 143k was already a big blip) - we get a bandwidth requirement of 343 MB/sec. Because tweets are going over TCP presumably and \~20% of a TCP connection is overhead, you would need 428 MB/s of bandwidth, about 3.5 gigabit or less than 0.5 of a 10 gigabit connection.
On the backend: lets assume we want triple redundancy (1 master + 2 replica) and that the average tweet goes out to 9 subscribers. This means that internally we need to write each tweet 30 times (we suppose a completely denormalized structure, we need to write the tweet to the users timeline also and do all this thrice for redundancy). This means 10 GB/sec of data (13 if we’re sending it over the network using TCP).
Thus \~100 servers would be able to easily handle the load. And remember this is 10x of the peak traffic they experienced.

So why do the have 20 to 40 times that many servers? This means that less than 10% (!) of their server capacity is actually used for business functions.

Second example: Google with DataStax came out with a blogpost about benchmarking a 300 node Cassandra cluster on Google Compute Engine. They claim a peak of 1.2M messages per second. Again, lets do some calculations:

The messages were 170 bytes in size. They were written to 2+1 nodes which would mean \~600 MB/s of traffic (730 MB/s if over the network using TCP).
They used 300 servers but were also testing the resiliency by removing 1/3 of the nodes, so lets be generous and say that the volume was divided over 100 servers.

This means that per server we use 7.3 MB/s network traffic and 6 MB/s disk traffic or 6% or a Gigabit connection and about 50% of medium quality spinning rust HDD.

My challenge to you is: next time you see such benchmarks do a quick back-of-the envelope calculation and if it uses less than 60% of the available throughput, call the people on it!

Proxying pypi / npm / etc for fun and profit!

Attila-Mihaly Balazs — Wed, 05 Feb 2014 17:26:00 +0200

Package managers for source code (like pypi, npm, nuget, maven, gems, etc) are great! We should all use them. But what happens if the central repository goes down? Suddenly all your continious builds / deploys fail for no reason. Here is a way to prevent that:

Configure Apache as a caching proxy fronting these services. This means that you can tolerate downtime for the services and you have quicker builds (since you don’t need to contact remote servers). It also has a security benefit (you can firewall of your build server such that it can’t make any outgoing connections) and it’s nice to avoid consuming the bandwidth of those registries (especially since they are provided for free).

Without further ado, here are the config bits for Apache 2.4

/etc/apache2/force_cache_proxy.conf - the general configuration file for caching:

# Security - we don't want to act as a proxy to arbitrary hosts
ProxyRequests Off
SSLProxyEngine On

# Cache files to disk
CacheEnable disk /
CacheMinFileSize 0
# cache up to 100MB
CacheMaxFileSize 104857600
# Expire cache in one day
CacheMinExpire 86400
CacheDefaultExpire 86400
# Try really hard to cache requests
CacheIgnoreCacheControl On
CacheIgnoreNoLastMod On
CacheStoreExpired On
CacheStoreNoStore On
CacheStorePrivate On
# If remote can't be reached, reply from cache
CacheStaleOnError On
# Provide information about cache in reply headers
CacheDetailHeader On
CacheHeader On

# Only allow requests from localhost

        Order Deny,Allow
        Deny from all
        Allow from 127.0.0.1


        # Don't send X-Forwarded-* headers - don't leak local hosts
        # And some servers get confused by them
        ProxyAddHeaders Off


# Small timeout to avoid blocking the build to long
ProxyTimeout    5

Now with this prepared we can create the individual configurations for the services we wish to proxy:

For pypi:

# pypi mirror
Listen 127.1.1.1:8001


        Include force_cache_proxy.conf

        ProxyPass         /  https://pypi.python.org/ status=I
        ProxyPassReverse  /  https://pypi.python.org/

For npm:

# npm mirror
Listen 127.1.1.1:8000


        Include force_cache_proxy.conf

        ProxyPass         /  https://registry.npmjs.org/ status=I
        ProxyPassReverse  /  https://registry.npmjs.org/

After configuration you need to enable the site (a2ensite) as well as needed modules (a2enmod - ssl, cache, disk_cache, proxy, proxy_http).

Finally you need to configure your package manager clients to use these endpoints:

For npm you need to edit ~/.npmrc (or use npm config set) and add registry = http://127.1.1.1:8000/

For Python / pip you need to edit ~/.pip/pip.conf (I recommend having download-cache as per Stavros’s post):

[global]
download-cache = ~/.cache/pip/
index-url = http://127.1.1.1:8001/simple/

If you use setuptools (why!? just stop and use pip :-)), your config is ~/.pydistutils.cfg:

[easy_install]
index_url = http://127.1.1.1:8001/simple/

Also, if you use buildout, the needed config adjustment in buildout.cfg is:

[buildout]
index = http://127.1.1.1:8001/simple/

This is mostly it. If your client is using any kind of local caching, you should clear your cache and reinstall all the dependencies to ensure that Apache has them cached on the disk. There are also dedicated solutions for caching the repositories (for example devpi for python and npm-lazy-mirror for node), however I found them somewhat unreliable and with Apache you have a uniform solution which already has things like startup / supervision implemented and which is familiar to most sysadmins.

Programming advent calendars for 2013

Attila-Mihaly Balazs — Wed, 04 Dec 2013 18:41:00 +0200

Programming advent calendars are posts/articles for a particular topic posted daily between the 1st and 24th of December. They are modeled on the advent calendars received by children on some countries which contain 24 doors for the 24 days of advent and behind each door is a piece of chocolate or other surprise which the child gets on the particular day.

Here is the list of programming related advent calendars for 2013 (if you know of more, leave a comment and I’ll update the list):

JavaAdvent

%0A

<a%20href=) - the Java Advent calendar - about all things in the JVM ecosystem (Scala, Clojure, Kawa, NetRexx!). I’m partial to this since I helped start this last year :-)

24 ways - “is the advent calendar for web geeks. Each day throughout December we publish a daily dose of web design and development goodness to bring you all a little Christmas cheer”

The Perl Advent Calendar 2013

Perl 6 Advent Calendar (if you didn’t hear - Perl 6 is a completely different language from Perl 5!)

SysAdvent (not only) for sysadmins

Performance Calendar Learn all about web performance. (Primarily focused on frontend technologies.)

UXmas UX is for everyone. Brush up on the topic this December.

Go Advent - the go(lang) advent calendar

24 Pull Requests - “a yearly initiative to encourage developers around the world to send a pull request every day in December up to Xmas”

Cleaning up Google AppEngine Mapreduce Jobs

Attila-Mihaly Balazs — Tue, 12 Nov 2013 12:31:00 +0200

Do you use the Google MapReduce library on AppEngine? And do you have a lot of completed tasks which clutter your dashboard? Use the JS below by pasting it into your developer console to clean them up! (use it at your own risk, no warranty is provided :-))

schedule = function() { window.setTimeout(function() { var c = $('a:contains(Cleanup)').first(); if (c.length > 0) { c.click(); } else { $('a:contains(Next page)').click(); schedule(); } }, 300); return true; }; window.confirm = schedule; schedule();

Capturing your screen on Ubuntu - with sound

Attila-Mihaly Balazs — Wed, 14 Aug 2013 09:50:00 +0300

Today I have a short script which I cobbled together from Google searches to do screen captures / screen casts with Ubuntu (including audio in so that you can narrate what is going on):

#!/bin/bash
Xaxis=$(xrandr -q | grep '*' | uniq | awk '{print $1}' |  cut -d 'x' -f1)
Yaxis=$(xrandr -q | grep '*' | uniq | awk '{print $1}' |  cut -d 'x' -f2)
avconv -f alsa -i pulse -f x11grab -s $(($Xaxis))x$(($Yaxis)) -i $DISPLAY.0 -r 15 -c:v libx264 -crf 0 -c:a libvo_aacenc -b:a 256k -threads 8 ~/Videos/output.mp4

I found this to work much better than gtk-recordmydesktop, which had lags, especially when “effects” were being drawn on the screen (like bulletpoints sliding in for a presentation or switching between desktops).

Tips for running SonarQube on large / legacy codebases

Attila-Mihaly Balazs — Fri, 28 Jun 2013 12:49:00 +0300

Crossposted from the Transylvania JUG website.

SonarQube (previously Sonar) is a quality management platform aimed mainly at Java (although other programming languages are supported to a varying degree. Here are a couple of tips to get it working on legacy projects:

There is an Ant runner and a standalone runner, it is not mandatory to use Maven (although it is a good idea in general to use it)
Look into the analysis parameters to customize it for your code.
Give it space and time :-). For reference a \~2 million LOC Java project took 77 minutes to be analyzed on my laptop (an Intel i7) with 4G heap.
To avoid having a ton of problems reported and to focus only on new problems, look into the Cutoff plugin
Test and coverage reports can be reused, no need to run them twice (once for the CI system and then for SonarQube). Look into reusing existing reports. Also, make sure to use the latest version of JaCoCo when generating profile data.
Configure your sonar.exclusions property to ignore code you aren’t interested in
Raise your sonar.findbugs.timeout property (the default of 5 minutes can be low for large projects)
Consider disabling source code related plugins (sonar.scm.enabled, sonar.scm-stats.enabled) if the provider for your SCM has an issue (HG has an issue currently for example with username containing spaces)

Keep your code clean!

Nested fluent builders

Attila-Mihaly Balazs — Sun, 23 Jun 2013 15:26:00 +0300

Crossposted from the Transylvania JUG website.

Builders have become commonplace in current Java code. They have the effect of transforming the following code:

“` {lang=”java” escaped=”true”} new Foo(1, 5, “abc”, false);

</code>

Into something like

``` {lang="java" escaped="true" line="1"}
Foo.builder()
  .count(1)
  .priority(5)
  .name("abc")
  .canonical(true)
  .build();

This has the advantage of being much easier to understand (as a downside we can mention the fact that - depending on the implementation - it can result in the creation of an additional object). The implementation of such builders is very simple - they a list of “setters” which return the current object:

“` {lang=”java” escaped=”true” line=”1”} public final class FooBuilder { private int count = 1; // …

public FooBuilder count(int count) { this.count = count; return this; }

public Foo build() { return new Foo(count, //… } }

</code>

Of course writing even this code can become repetitive and annoying, in
which case we can use
[Lombok](http://projectlombok.org/features/experimental/Builder.html) or
other code generation tools. An other possible improvement - which makes
builder more useful for testing - is to add methods like random as
suggested in [this Java Advent
Calendar](http://www.javaadvent.com/2012/12/using-builder-pattern-in-junit-tests.html)
article. We can subclass the builder (into FooTestBuilder for example)
and only use the "extended" version in testing.

What can do however if our objects are more complex (they have
non-primitive fields)? One approach may look like this:

``` {lang="java" escaped="true" line="1"}
Foo.builder()
  .a(1)
  .b(2)
  .bar(Bar.builder().c(1).build())
  .buzz(Buzz.builder().build())
  .build();

We can make this a little nicer by overloading the bar / buzz methods to accept instances of BarBuilder / BuzzBuilder, in which case we can omit two build calls. Still, I longed for something like the following:

“` {lang=”java” escaped=”true” line=”1”} Foo.builder() .a(1) .b(2) .bar() .c(1).build() .buzz() .build() .build();

</code>

The idea is that the bar / buzz calls call start a new "context" where
we initialize the Bar/Buzz classes. "build" calls end the innermost
context, with the last build returning the initialized Foo object
itself. How can this be written in a typesafe / compiler verifiable way?

My solution is the following:

-   Each builder is parameterized to return an arbitrary type T from its
    build method
-   The actual return value is generated from a Sink of T
-   When using the builder at the top level, we use an IdentitySink with
    just returns the passed in value.
-   When using the builder in a nested context, we use a Sink which
    stores the value and returns the builder from "one level up".

Some example code to clarify the explanation from above can be found
below. Note that this code has been written as an example and could be
optimized (like making using a single instance of the IdentitySink,
having FooBuilder itself implementing the sink methods, etc).

Implementation of a leaf-level builder:

``` {lang="java" line="1"}
interface Sink {
  T setBar(Bar bar);
}

final class Bar {
  // ...
  public static BarBuilder builder() {
    return new BarBuilder(new Sink() {
      @Override
      public Bar setBar(Bar bar) { return bar; }
    });
  }
}

class BarBuilder {
  // ...

  protected BarBuilder(Sink sink) {
    this.sink = sink;
  }

  // ...

  public T build() {
    return sink.setBar(new Bar(c, d, fizz));
  }
}

Implementation of the root level builder:

“` {lang=”java” line=”1”} class FooBuilder { // … public BarBuilder setBar() { return new BarBuilder(new Sink() { @Override public Bar setBar(Bar bar) { FooBuilder.this.bar = bar; return FooBuilder.this; } }); }

// … } “`

Conclusion: Java has some missing features (liked named parameters or the ease of reuse provided by duck-typing). We can work around them however nicely with some carefully crafted code (and we can put repeating code into code generators to avoid having to write it over and over again). In exchange we get a very versatile and good performing cross-platform runtime.

Passing UTF-8 trough HTTP

Attila-Mihaly Balazs — Wed, 29 May 2013 20:36:00 +0300

These days we should write every code as if it will be used by international people with a wide variety of personal information (just look at Falsehoods Programmers Believe About Names for some headscratchers). I would like to do add my small contribution to this by showing how UTF-8 encoded strings can be passed into GET/POST parameters.

For this I’ll be using the following small PHP script, which can be quickly run by the command line PHP webserver added in PHP 5.4:

GETs: 
POSTs:

We’ll test this with the following Python script:

#!/usr/bin/python
# vim: set fileencoding=utf-8 :
import urllib
import urllib2

params = {'name': u'東京'}
params = { k: v.encode('utf-8') for k, v in params.iteritems() }
data = urllib.urlencode(params)

url = 'http://localhost:8000/?' + data
req = urllib2.Request(url, data)
response = urllib2.urlopen(req)

print response.read()

This all works well and nicely, so here are some conclusions:

GET and POST variables need to be UTF-8 encoded after which they need to be urlencoded (“% encoded”). See this StackOverflow answer.
Based on the same answer: hostnames use Punycode instead (but we are not concerned with hostnames here)
You might need to add the following header for POST requests to work: “Content-Type: application/x-www-form-urlencoded; charset=UTF-8”
Failing to observe this sequence leads to an UnicodeEncodeError in urllib.urlencode

Connecting to the MtGox market data feed using Perl

Attila-Mihaly Balazs — Wed, 29 May 2013 17:52:00 +0300

For a recent project I needed some realistic market data for an electronic exchange. Seeing how MtGox provides free and open access to theirs (thank you!) I chose them. However none of the examples floating around the internet seemed to work, so I whipped one up using Net::Async::WebSocket::Client. Enjoy:

use IO::Async::Loop;
use Net::Async::WebSocket::Client;

my $client = Net::Async::WebSocket::Client->new(
        on_frame => sub {
                my ( $self, $frame ) = @_;
                print "\n", $frame, "\n";
        },
);

my $loop = IO::Async::Loop->new;
$loop->add( $client );

$client->connect(
        host => 'websocket.mtgox.com',
        service => 80,
        url => "ws://websocket.mtgox.com:80/mtgox",
        on_connected => sub {},
        on_connect_error => sub { die "Cannot connect - $_[-1]" },
        on_resolve_error => sub { die "Cannot resolve - $_[-1]" },
);

$loop->loop_forever;

(it is basically the sample program for the module, with the MtGox market data URL hardcoded).

Converting datetime to UTC in python

Attila-Mihaly Balazs — Thu, 07 Feb 2013 19:06:00 +0200

So you need to convert a python datetime object which has a timezone set (“aware” in the Python nomenclature) to an UTC one with no timezone set (“naive”), for example because NDB on GAE can’t store anything else. The solution will look something like this:

date = date.astimezone(tz.tzutc()).replace(tzinfo=None)

For searcheability: the exception thrown by NDB if you fail to do this is “NotImplementedError: DatetimeProperty updated_at can only support UTC. Please derive a new Property to support alternative timezones.”

Recovering your RoTLD password for domains registered trough Gandi.NET

Attila-Mihaly Balazs — Tue, 05 Feb 2013 16:08:00 +0200

If you need to “recover” your RoTLD password when the .RO domain is registered trough Gandi.NET (I say “recover” because you didn’t set it in the first place :-)) - do this:

In the Gandi interface go to Account Management -> Update account information and set “Anti-spam system” to No
Go to the RoTLD password recovery page and reset your password. Now the password recovery email should have arrived to your inbox.
(Optional) go back to the Gandi interface and re-enable the anti-spam system

I needed to do this to enable CloudFlare on IT-Events.RO because RoTLD allows the changing of nameservers only trough their website.

Free software for Windows

Attila-Mihaly Balazs — Fri, 04 Jan 2013 01:36:00 +0200

Inspired by this post I decided to list the software I use/recommend under Windows.

Free/Libre Open Source Software:

LibreOffice (for of OpenOffice) - a very capable office solution. Most people don’t need anything more than this. If you are installing it for a non-technical user, make sure to set the default file formats to the Microsoft ones (.doc, .xls, …)
Far Manager - a very cool two-panel file manager, for those of us who like text-based things (don’t let the looks fool you - it is very modern and very capable). You could also take a look at NDN or DNOSP, but FAR is my personal favorite.
VLC - THE video player. It can play 99.999% of the media out there and it won’t pollute your system with all kinds of DLLs.
ffdshow-tryouts - DirectShow / VFW codecs for all the formats VLC can play (in fact they are both based on the ffmpeg project). Use this to play back videos in programs which are DirectShow based (like WMP or Winamp).
7-zip - for all your zipping and unzipping needs. It supports a lot of other formats too (mainly for extracting) so no need to install the shareware version of WinRar
Firefox - ‘nuff said. There are also a lot of plugins which one might find useful like Firebug, NoScript, RequestPolicy, etc. As a download manager I would recommend DownThemAll, but I found that with recent increases in Internet access speeds I don’t need a download manager.
Notepad2 for your small (text) editing needs. There is also Notepad++ and notepad2-mod.
PDFForge to create PDFs from any program which can print (it acts as a virtual printer which outputs its results to a PDF file). Sidenote: LibreOffice can natively save to PDF, no need for this if you’re using it only for that.
Pidgin - you might know it as GAIM, a multi protocol instant messenger. And it is very multi protocol. The only downside is that some advanced protocol features are not always functional (*cough* file-transfer, *cough*), but I find that I rarely use those anyways. If you are installing it for someone else however, make sure to ask them what features they consider essential (like custom background/emoticons) and act accordingly.
WinSCP to copy files trough SSH/SCP, and the FileZilla client to do the same over FTP.
Various specialized programs: GIMP for photo-editing, Inkscape for vector-based graphics, VirtualDub and Avidemux for (linear) video editing, Wireshark for network analysis, Audacity for audio editing, PuTTY as an SSH client, VirtualBox for running virtual machines and Eclipse for development (it can do more than Java!). There is also the IntelliJ Community Edition for developement which is Open Source Software, but be aware of the limitations.
XAMPP for quickly setting up a LAMP environment under Windows

Free (as in beer) - these may try to trick you into installing toolbars / changing your homepage / your search engine so watch out:

TeamViewer - a nice remote control solution (also cross platform - although it doesn’t run perfectly on other OSs). I especially like the fact that it can run as a service and that it takes care of the NAT traversal problem.
CDBurnerXP - for burning optical media. Nothing special, but it works.
IrfanView - a very capable image viewer / converter. Don’t forget to install the plugins to take full advantage of its features! It is also so lightweight that won’t believe how quickly the installation finishes. Watch out though, it tries to install sponsored programs :-(
FreeCommander - a two panel graphic file manager. Recommended if you’re a Total/Windows Commander fan rather than a Norton Commander one :-)
The MS Visual Studio Express series - a good way to get your feet wet with MS specific development (also good for university projects), but be aware that you’ll quickly hit a wall with it on professional projects.
uTorrent for my downloading needs.
Daemon Tools Free for my ISO (CD/DVD/BR) mounting needs. Attention during install! It will try to “upgrade” you several times during install and also try to install additional software / change your home page / search provider if you just click trough next. Don’t let it, form a technical point of view it is a great product. There is also the unsupported Virtual CD product from Microsoft and Windows can mount ISOs natively starting from Windows 7 I think.
The FoxIt PDF Reader - a lightweight PDF reader, although Adobe Reader X caught up nicely I feel (and they also auto-update to eliminate security vulnerabilities), so you could give it a second go.
BB FlashBack Express for screen recording. Little annoying to install (you need to give them an email account, they try to upgrade you to the paid version and you need to “register” afterwards), but after installing it is all good and I found it to be a very capable product (even at the free version level).
Chrome and Opera as alternative browsers (and no, Chrome is not Open Source, Chromium is).
Paint.NET for advanced but “not photoshop level” image editing.
foobar2000 or Winamp (with the classic skin :-)) for music. foobar is very lightweight and quick, but it might lack some features. Winamp is very complete, but tries to make all kinds of changes to your system. You also probably don’t need the Winamp Agent to run in the background all the time :-)
Dropbox for file synchronization / small-scale file serving. Alternatively there is SkyDrive, but it isn’t very Linux friendly.
Windows Live Writer - the best blog publishing software I could find. Unfortunately Microsoft ruined it with the Office 2007 look and now seems to want to abandon it completely
Skype for video-call / teleconference, although lately I’ve been dropping it in favor of Google Hangouts

Update: this software looks very promising: Ninite. It purports to auto-install and auto-update a lot of common Windows software. Will take it for a spin the next time I install Windows.

What every programmer should know about X

Attila-Mihaly Balazs — Mon, 31 Dec 2012 11:24:00 +0200

Piggybacking on some memes floating around on the internet I would like to publish my list of “what every programmer should know”.

A couple of introductory words: in my opinion the two most important things to learn for new programmers are terminology - to know what things / ideas / algorithms / concepts are called so that they can search for them on the internet and discuss their ideas) and humility (if something doesn’t exists or doesn’t work the way we expected, the first thing we should ask ourselves is: “what am I missing?” instead of proclaiming the predecessors to be idiots). Moving along to the list:

What every programmer should know about X - the Breaking Eggs And Making Omelettes blog contains a very eclectic list of articles. The only thing I would add is the PDF link to “What every programmer should know about memory” rather than the LWN version. Also worth checking out the discussion in the comments.
Latency numbers every programmer should know and the discussion on hackernews (this is not the original source of the data, but a particularly good visualization of it nevertheless - see also my extended version which allows you to transform the numbers into whichever time-unit - s, ms, us - you need).
From the “Falsehoods programmers believe about X” department comes:
- Falsehoods programmers believe about build systems (additionally to the comments on the post, also see the discussion on Reddit and a follow-up blogpost)
- Falsehoods Programmers Believe About Names
- Falsehoods programmers believe about time and More falsehoods programmers believe about time
A video giving an overview of the different Google systems (search, map-reduce, GFS, etc). Nothing revolutionary, but a good first-time overview:

Happy holiday reading/watching to all!

Ensuring the order of execution for tasks

Attila-Mihaly Balazs — Fri, 21 Dec 2012 21:13:00 +0200

This post was originally published as part of the Java Advent series. If you like it, please spread the word by sharing, tweeting, FB, G+ and so on! Want to write for the Java Advent blog? We are looking for contributors to fill all 24 slot and would love to have your contribution! Contact Attila Balazs to contribute!

Sometimes it is necessary to impose certain order on the tasks in a threadpool. Issue 206 of the JavaSpecialists newsletter presents one such case: we have multiple connections from which we read using NIO. We need to ensure that events from a given connection are executed in-order but events between different connections can be freely mixed.

I would like to present a similar but slightly different situation: we have N clients. We would like to execute events from a given client in the order they were submitted, but events from different clients can be mixed freely. Also, from time to time, there are “rollup” tasks which involve more than one client. Such tasks should block the tasks for all involved clients (but not more!). Let’s see a diagram of the situation:

![](http://1.bp.blogspot.com/-TuCHb25JBqM/UNOOV5-EjwI/AAAAAAAAFo4/KoIJZOXz2Y8/s320/Untitled%2Bdrawing%2B%25281%2529.png)

As you can see tasks from client A and client B are happily processed in parallel until a “rollup” task comes along. At that point no more tasks of type A or B can be processed but an unrelated task C can be executed (provided that there are enough threads). The skeleton of such an executor is available in my repository. The centerpiece is the following interface:

public interface OrderedTask extends Runnable {
    boolean isCompatible(OrderedTask that);
}

Using this interface the threadpool decides if two tasks may be run in parallel or not (A and B can be run in parallel if A.isCompatible(B) && B.isComaptible(A)). These methods should be implemented in a fast, non locking and time-invariant manner.

The algorithm behind this threadpool is as follows:

If the task to be added doesn’t conflict with any existing tasks, add it to the thread with the fewest elements.
If it conflicts with elements from exactly one thread, schedule it to be executed on that thread (and implicitly after the conflicting elements which ensures that the order of submission is maintained)
If it conflicts with multiple threads, add tasks (shown with red below) on all but the first one of them on which a task on the first thread will wait, after which it will execute the original task.

![](http://4.bp.blogspot.com/-IcwKi0hwcyA/UNSsiYecUPI/AAAAAAAAFpU/zk41QVcuxLo/s320/Untitled%2Bdrawing%2B%25282%2529.png)

More information about the implementation:

The code is only a proof-of-concept, some more would would be needed to make it production quality (it needs code for exception handling in tasks, proper shutdown, etc)
For maximum performance it uses lock-free* structures where available: each worker thread has an associated ConcurrentLinkedQueue. To achieve the sleep-until-work-is-available semantics, an additional Semaphore is used**
To be able to compare a new OrderedTask with currently executing ones, a copy of their reference is kept. This list of copies is updated whenever new elements are enqueued (this is has the potential of memory leaks and if tasks are infrequent enough alternatives - like an additional timer for weak references - should be investigated)
Compared to the solution in the JavaSpecialists newsletter, this is more similar to a fixed thread pool executor, while the solution from the newsletter is similar to a cached thread pool executor.
This implementation is ideal if (a) the tasks are (mostly) short and (mostly) uniform and (b) there are few (one or two) threads submitting new tasks, since multiple submissions are mutually exclusive (but submission and execution isn’t)
If immediately after a “rollup” is submitted (and before it can be executed) tasks of the same kind are submitted, they will unnecessarily be forced on one thread. We could add code rearrange tasks after the rollup task finished if this becomes an issue.

Have fun with the source code! (maybe some day I’ll find the time to remove all the rough edges).

* somewhat of a misnomer, since there are still locks, only at a lower - CPU not OS - level, but this is the accepted terminology

** - benchmarking indicated this to be the most performant solution. This was inspired from the implementation of the ThreadPoolExecutor.

Meta: this post is part of the Java Advent Calendar and is licensed under the Creative Commons 3.0 Attribution license. If you like it, please spread the word by sharing, tweeting, FB, G+ and so on! Want to write for the blog? We are looking for contributors to fill all 24 slot and would love to have your contribution! Contact Attila Balazs to contribute!

Java Runtime options

Attila-Mihaly Balazs — Fri, 21 Dec 2012 21:06:00 +0200

The Java runtime is a complex beast - and it has to be since it runs officially on seven platforms and unofficially on many more. Give this, it is normal that there are many knobs and dials to control how things function. The more well known ones are:

-Xmx for the maximum heap size
-client and -server for selecting the default set of parameters from classes of defaults
-XX:MaxPermGen for controlling the permanent generation size

Other than these, it is (very) rarely the case that you need to change the defaults. However, thanks to Java being open source you can see the list of options, their default values and a short explanation directly from the source code. Currently there are almost 800 options in there!

An other way to see the options (but one which doesn’t display the explanations unfortunately) is the following command:

java -XX:+UnlockDiagnosticVMOptions -XX:+UnlockDiagnosticVMOptions -XX:+PrintFlagsFinal -version

These options are well worth studying. Not for tweaking them (since there is a wealth of testing behind the defaults the extent of which would be very hard to replicate), but rather to understand the different functionalities offered by the JVM (for example why you might not see stacktraces in exceptions).

Changes to String.substring in Java 7

Attila-Mihaly Balazs — Wed, 12 Dec 2012 10:37:00 +0200

It is common knowledge that Java optimizes the substring operation for the case where you generate a lot of substrings of the same source string. It does this by using the (value, offset, count) way of storing the information. See an example below:

![](http://4.bp.blogspot.com/-gnaLPXGMeUQ/UMIaKhQ5wsI/AAAAAAAAFn8/wNPgGPtE2qY/s320/Untitled%2Bdrawing.png)

In the above diagram you see the strings “Hello” and “World!” derived from “Hello World!” and the way they are represented in the heap: there is one character array containing “Hello World!” and two references to it. This method of storage is advantageous in some cases, for example for a compiler which tokenizes source files. In other instances it may lead you to an OutOfMemorError (if you are routinely reading long strings and only keeping a small part of it - but the above mechanism prevents the GC from collecting the original String buffer). Some even call it a bug. I wouldn’t go so far, but it’s certainly a leaky abstraction because you were forced to do the following to ensure that a copy was made: new String(str.substring(5, 6)).

![](http://3.bp.blogspot.com/-NGSJS_psCIc/UMW40g0NLXI/AAAAAAAAFoQ/7kfOVA8JdC0/s320/Untitled%2Bdrawing.png)

This all changed in May of 2012 or Java 7u6. The pendulum is swung back and now full copies are made by default. What does this mean for you?

For most probably it is just a nice piece of Java trivia
If you are writing parsers and such, you can not rely any more on the implicit caching provided by String. You will need to implement a similar mechanism based on buffering and a custom implementation of CharSequence
If you were doing new String(str.substring) to force a copy of the character buffer, you can stop as soon as you update to the latest Java 7 (and you need to do that quite soon since Java 6 is being EOLd as we speak).

Thankfully the development of Java is an open process and such information is at the fingertips of everyone!

A couple of more references (since we don’t say pointers in Java :-)) related to Strings:

If you are storing the same string over and over again (maybe you’re parsing messages from a socket for example), you should read up on alternatives to String.intern() (and also consider reading chapter 50 from the second edition of Effective Java: Avoid strings where other types are more appropriate)
Look into (and do benchmarks before using them!) options like UseCompressedStrings (which seems to have been removed), UseStringCache and StringCache

Hope I didn’t strung you along too much and you found this useful! Until next time
- Attila Balazs

Meta: this post is part of the Java Advent
Calendar and is licensed under the Creative Commons 3.0 Attribution license. If you like it, please spread the word by sharing, tweeting, FB, G+ and so on! Want to write for the blog? We are looking for contributors to fill all 24 slot and would love to have your contribution! Contact Attila Balazs to contribute!

(Re)Start me up!

Attila-Mihaly Balazs — Sat, 01 Dec 2012 18:24:00 +0200

This post was originally published as part of the Java Advent series.

There are cases where you would like to start a Java process identical to the current one (or at least using the the same JVM with tweaked parameters). Some concrete cases where this would be useful:

Auto-tuning the maximum memory parameters (ie. you have an algorithm to determine the optimal value - for example: 80% of the system memory - and your JVM wasn’t started with that particular value)
Creating a cluster of processes for high(er)-availability (true HA implies multiple physical nodes) or because processes have different roles (like the components in MongoDB).
Daemonizing the current process (that is, the background process should run even after the launching process has terminated) - this is a very frequent modus-operandi for programs on *nix systems where you have the foreground “control” process and the background “daemon” process (not to be confused with the “daemon” threads).

Doing this is relatively simple - and can be done in pure Java - after you find the correct API calls:

“` {style=”overflow: auto;”} List arguments = new ArrayList(); // the java executable arguments .add(String.format(“%s%sbin%sjava”, System.getProperty(“java.home”), File.separator, File.separator)); // pre-execuable arguments (like -D, -agent, etc) arguments.addAll(ManagementFactory.getRuntimeMXBean() .getInputArguments());

String classPath = System.getProperty(“java.class.path”), javaExecutable = System .getProperty(“sun.java.command”); if (classPath.equals(javaExecutable)) { // was started with -jar arguments.add(“-jar”); arguments.add(javaExecutable); } else { arguments.add(“-classpath”); arguments.add(classPath); arguments.add(javaExecutable); }

// we might add additional arguments here which will be received by the // launched program // in its args[] paramater arguments.add(“runme”);

// launch it! new ProcessBuilder().command(arguments).start(); “`

Some explanations about to the code:

It is largely inspired from this project
We suppose that the java executable is named java and is located in bin/java relative to java.home. We use File.separator for the code to be portable.
getInputArguments is used to get specific arguments passed to the JVM (like -Xmx). It does not include the classpath.
Which is taken from java.class.path
Finally, there is one heuristic step: we try to detect if we were launched using the -jar myjar.jar syntax or the MyMainClass syntax and replicate it.

This is it! After that we use ProcessBuilder (which we should always favour over Runtime.exec because it auto-escapes the parts of the command line for us).

A final thought: if you intend to use this method to “daemonize” a process (that is: to ensure that it stays running after its parent process has terminated) you should do two things:

Redirect the standard input and output. By default they are redirected into temporary buffers and the JVM will seemingly randomly terminate when those buffers (pipes) fill up.
Under Windows use javaw instead of java. This ensures that the process won’t be tied to the console it was started from (however it will still be tied to the user login session and will terminate when the user logs out - for a more heavy-duty solution look into the Java Service Wrapper).

This is it for today, hope you enjoyed it, fond it useful. If you run the code and it doesn’t work as advertised, let me know so that I can update it (I’m especially interested if it works with non Sun/Oracle JVMs). Come back tomorrow for an other article!

Meta: this post is part of the Java Advent Calendar and is licensed under the Creative Commons 3.0 Attribution license.

Upgrading from MySQL to MariaDB on Ubuntu

Attila-Mihaly Balazs — Sun, 25 Nov 2012 11:14:00 +0200

So you decided that Oracle doesn’t know its left foot from the back of his neck when it comes to open source (how’s that for a mixed metaphor), but you are not ready just yet to migrate over to PostgreSQL? Consider MariaDB. Coming from Monty Widenius, the original author of MySQL, it aims to be 100% MySQL compatible while also being truly open-source.

Give that it’s 100% MySQL compatible, you can update in-place (nevertheless it is recommended that do a backup of your data first). The steps are roughly adapted from here.

Go to the MariaDB repository configuration tool and generate your .list file (wondering what’s up with the 5.5 vs 10.0 version? See this short explanation). You don’t know the exact Ubuntu version you’re running? Just use lsb_release -a.
Save the generated file under /etc/apt/sources.list.d/MariaDB.list as recommended and do an sudo aptitude update. You should see an output complaining about some public keys.
Do sudo apt-key adv --recv-keys --keyserver keyserver.ubuntu.com 0xCBCB082A1BB943DB to add those keys (replace the last number with the one you saw in the previous output).
Issue sudo apt-cache policy mysql-common and you should see mariadb as an upgrade option.
Finally do sudo aptitude upgrade mysql-common libmysqlclient18 and watch your MySQL database being transformed into a MariaDB one and all keeping chugging along just as usual!

Cluj-Napoca (Romania) wins the title of European Youth Capital 2015

Attila-Mihaly Balazs — Sun, 25 Nov 2012 10:58:00 +0200

We interrupt our regular (lack of) posting to bring you this news:

![](http://4.bp.blogspot.com/-cJh3GYVhtbA/ULHdZ2-l8pI/AAAAAAAAFkU/CeHgOqxbKhA/s320/18089_10151270244849851_235858203_n.jpg)

Cluj-Napoca is European Youth Capital 2015. Congratulation to everyone involved!

Writing beautiful code - not just for the aesthetic value

Attila-Mihaly Balazs — Tue, 06 Nov 2012 14:33:00 +0200

This article was originally published in the 6th edition of TodaySoftMag in Romanian and on the Transylvania JUG blog in English. Reprinted here with the permission of the author / magazine.

Most mainstream programming languages contain a large set of features and diverse standard libraries. Because of this it becomes important to know not only “how” you can achieve something (to which there are usually several answers) but also “what is the recommended way”.

In this article I will argue that knowing and following the recommended ways of coding doesn’t only yield shorter (easier to write), easier to read, understand and maintain code but also prevents programmers from introducing a lot of bugs.

This particular article needs a drop of Java language knowledge to savour, but the fundamental idea can be generalized to any programming language: there is more to using a language efficiently than just knowing the syntax.

Example 1: Double Trouble

Lets start with a snippet of code: what does it print out?

Double d1 = (5.0d - 5.0d) *  1.0d;
Double d2 = (5.0d - 5.0d) * -1.0d;
System.out.println(d1.equals(d2));

What about the following one?

double d1 = (5.0d - 5.0d) *  1.0d;
double d2 = (5.0d - 5.0d) * -1.0d;
System.out.println(d1 == d2);

The answer seems to be clear: in both cases we multiply zero with different values (plus and minus one respectively), thus the result should be zero which should compare as equal regardless of the comparison method used (calling the equals method on objects or using the equality operator on the primitive values).

If we run the code, the result might surprise us: the first one displays false while the second one displays true. What’s going on? On one level we can talk the technical reasons behind this result: floating point values are represented in Java (and many other programming languages) using the sign-and-magnitude notation defined in the IEEE Standard 754. Because of this technical detail both “plus zero” and “minus zero” can be represented by variables of this type. And the “equals” method on Double (and Float) objects in Java considers these values to be distinct.

On another level however we could have avoided this problem entirely by using the primitive values as shown in the second code snippet and as suggested by Item 49 in the Effective Java book^[1]^: Prefer primitive types to boxed primitives. Using primitive types is also more memory efficient and saves us from having to create special cases for the null value.

Sidenote: we have a similar situation with the BigDecimal class^[2]^ where values scaled differently don’t compare as equal. For example the following snippet also prints false:

BigDecimal d1 = new BigDecimal("1.2");
BigDecimal d2 = new BigDecimal("1.20");
System.out.println(d1.equals(d2));

The answer in this case (given that there is no primitive equivalent for this class) would be to use the compareTo method and assert that it returns zero instead of using the equals method (a method which can also be used to solve the conundrum in the Double/Float case if we are not worried about nulls).

Example 2: Where is my null at?

What does the following snippet of code print out?

Double v = null;
Double d = true ? v : 0.0d;
System.out.println(d);

At first glance we would say: null, since the condition is true and v is null (and null can be assigned to a reference of any type, so we are allowed to use it). The actual result is however a NullPointerException at the second line. This is because the right-hand type of the assignment is actually double (the primitive type) not Double (as we would expect) which is silently converted into Double (the boxed type). The generated code looks like this:

Double d = Double.valueOf(true ? v.doubleValue() : 0.0d);

This behavior is described in the Java Language Specification^[3]^:

“If one of the second and third operands is of primitive type T, and the type of the other is the result of applying boxing conversion (§5.1.7) to T, then the type of the conditional expression is T.”

I would venture to guess that not many of us have read the JLS in its entirety and even if we would have read it, we might not have realized the implications of each phrase. The recommendation from EJ2nd mentioned at the previous example saves us again: we should use primitive types. We can also draw a parallel with Item 43: Return empty arrays or collections, not nulls. Would we have used a “neutral element”, which is analogous to using empty arrays/collections, the problem would not have appeared. (The neutral element would be 0.0d if we use the value later in summation or 1.0d if we use it in multiplication.)

Example 3: We come up empty

What is the difference between the following two conditions?

Collection items;
if (items.size() == 0) { ... }
if (items.isEmpty()) { ... }

One could argue that they do exactly the same thing as being empty is equivalent to having zero items. Still, the second condition is easier to understand (we can almost read it out loud: “if items is empty then …”). But there is more: in some cases it can be much, much faster. Two examples from the Java standard libraries where the time needed to execute “size” grows linearly with the number of elements in the collection while “isEmpty” returns in constant time: ConcurrentLinkedQueue^[4]^ and the view sets returned by TreeSet’s^[5]^ headSet/tailSet methods. And while the documentation for the first mentions this fact, it doesn’t for the second.

This is yet another example how nicer code is also faster.

Example 4: Careful with that static, Eugene!

What will the following snippet of code print out?

public final class Test {
        private static final class Foo {
                static final Foo INSTANCE = new Foo(); // 2
                static final String NAME = Foo.class.getName(); // 3
                Foo() {
                        System.err.println("Hello, my name is " + NAME);
                }
        }
        public static void main(String[] args) {
                System.err.println("Your name is what?\nYour name is who?\n");
                new Foo(); // 1
        }
}

It will be

Your name is what?
Your name is who?

Hello, my name is null
Hello, my name is Test$Foo

The (probably) unexpected null value happens because we obtain a reference to a partially constructed object:

We start to create an instance of Foo at point 1
This being the first reference to Foo, the JVM loads it and starts to initialize it
Initializing Foo involves initializing all its static fields
The initialization of the first static field contains a call to the constructor at point 2 which is dutifully executed
At this point the NAME static field is not yet initialized, so the constructor will print out null

This code demonstrates that static fields can be confusing and we shouldn’t use them for things other than constants (but even then we should evaluate if the constant is not better declared as an Enum). By the same token we should also avoid singletons which make our code harder to test (thus avoiding them will make the code easier to test).

We should however favor static member classes over non-static ones (Item 22 in EJ2nd). Static classes in Java are entirely distinct conceptually from static fields and it is unfortunate that the same word was used to describe them both.

We should also run static analysis tools on our code and verify their output frequently (ideally at every commit). For example the bug presented is caught by Findbugs^[6]^ and tools incorporating Findbugs.

Example 5: Remove old cruft

Name four things wrong with the following snippet:

// WRONG! DON’T DO THIS!
Vector v1;
...
if (!v1.contains(s)) { v1.add(s); }

They would be:

The wrong container type is used. We clearly want to have each string present at most once which suggests using a Set
which has the benefits of shorter and faster code (the above method gets linearly slower with the number of elements)
Doesn’t use generics
It unnecessarily synchronizes access to the structure if it is only used from a single thread
If the structure is actually used from multiple threads, the code is not thread safe, only “exception safe” (as in: no exceptions will be raised, but the data structure can be silently corrupted possibly creating a lot of headache downstream)

All of these can be avoided by dropping Vector and its siblings (Hashtable, StringBuffer) and using the Java Collection Framework (available for 14 years^[7]^) with generics (available for 8 years^[8]^).

Conclusion

There are many more examples one could give, but I think the point is well made that knowing a programming language means more than just knowing the syntax at a basic level. I’m urging you if you are using Java: get yourself a copy “Effective Java, 2nd edition” and “Java™ Puzzlers: Traps, Pitfalls, and Corner Cases” each and read through them if you haven’t done so already. Also, use static analysis on your code (Sonar^[9]^ is a good choice in this domain) and consider fixing the issues signaled by it, or at least read up on them.

Again, the conclusions is similar for other languages:

Try reading up on best practices/idiomatic ways to write code in the given language. For example for Perl the best book currently is “Modern Perl^[10]^” by chromatic
Look to see if there is a good quality static analysis / lint program for your language. For Perl there is Perl::Critic^[11]^, for Python there is pep8^[12]^ and pylint^[13]^, all of which are free and open source

Being good a programmer (or an architect, or a business analyst, etc) is process of lifelong learning and these are the tools which can help us truly learn a programming language.

[[1]](#ftnt_ref1)Joshua Bloch: Effective Java, Second Edition. ISBN: 0321356683

[[2]](#ftnt_ref2)

[[3]](#ftnt_ref3)

[[4]](#ftnt_ref4)

[[5]](#ftnt_ref5)

[[6]](#ftnt_ref6)

[[7]](#ftnt_ref7)

[[8]](#ftnt_ref8)

[[9]](#ftnt_ref9)

[[10]](#ftnt_ref10)

[[11]](#ftnt_ref11)

[[12]](#ftnt_ref12)

[[13]](#ftnt_ref13)

A (mostly) cross-platform clickable map of Romanian counties

Attila-Mihaly Balazs — Thu, 01 Nov 2012 10:06:00 +0200

[![](http://2.bp.blogspot.com/-efBtXhgduhw/UJItSwviTnI/AAAAAAAAFjw/-D_TsgQlsUs/s320/Screenshot%2Bfrom%2B2012-11-01%2B10%253A02%253A31.png)](http://hype-free.googlecode.com/svn/trunk/map-romania/index.html)

TL;DR - I’ve created a map of Romania’s counties (judete) using Javascript which can be used as an alternative for drop-down boxes during form input. It works great with FF and Chrome :-). See it in action and browse the source code.

The map is based on the SVG drawings from Wikimedia Commons postprocessed in Inkscape to simplify the paths (and to remove the Black Sea from it). After that it was converted to Raphael.js using readysetraphael. I also use ScaleRaphael to be able to render it at an arbitrary size. According to BrowserStack (a great service well worth its money BTW if you do web development!) it works with Firefox (starting with version 3.6), Chrome and Safari. IE has a hard time with it, but hopefully as the underlying library improves, it will start to work :-).

Every end is a new beginning

Attila-Mihaly Balazs — Sat, 27 Oct 2012 23:03:00 +0300

TL;DR: I’m shutting down the twitfeeder project (it was on lifesupport for a long time) so I mirrored a technical article from the blog in the hope that it might be useful for somebody someday.

Proxying URL fetch requests from the Google App Engine

Hosting on the Google App Engine means giving up a lot controls: your
application will run on some machines in one of Google’s datacenters,
use some other machines to store data and use yet an other set of
machines to talk to the outside world (send/receive emails, fetch URLs,
etc). You don’t know the exact identity of these machines (which for
this article means their external IP address), and even though you can
find out some of the details (for example by fetching the ipchicken page
and parsing the result or by sending an e-mail and looking at the
headers), there is no guarantee made that the results are repeatable
(ie. if you do it again and again you will get the same result) in the
short or long run. While one might not care about such details most of
the time, there are some corner cases where it would be nice to have a
predictable and unique source address:

You might
worry that some of the exit points get blocked because other
applications on the Google App Engine have an abusive behavior
You might want a single exit point so that you can “debug” your traffic at a low (network / tcpdump) level
And finally, the main reason for which the Twit Feeder uses it: some third-party services (like Twitter or Delicious) use the source IP to whitelist requests to their API

To
be fair to the GAE Architects: the above considerations don’t affect
the main usecase and are more of a cornercase if we look at the average
application running on the GAE. Also, having so few commitments (ie.
they don’t stipulate things like “all the URL fetches will come from the
10.0.0.1/24 netblock”) means that they are free to move things around
(even between datacenters) to optimize uptime and performance which in
turn benefits the application owners and users.

Back to
our scenario: the solution is to introduce an intermediary which has a
static and well known IP and let it do all the fetching. Ideally I would
have installed Squid and and
be done with it, but the URL fetch service doesn’t have support for
proxies currently. So the solution I came up with looks like this:

Get
a VPS server. I would recommend one which gives you more bandwidth
rather than more CPU / memory / disk. It is also a good idea to get one
in the USA to minimize latency from/to the Google datacenters. I’m
currently using VPSLink and I didn’t had any problems (full disclosure: that is a referral link and you should get 10% off for lifetime if you use it).
Install Apache + PHP on it
Use a simple PHP script to fetch the page encoded in a query parameter using the php_curl extension.

To spice up this blogpost ;-), here is a diagram of the process:

![](https://lh4.ggpht.com/_hrvCBhtWhJ4/S1IPu9C6clI/AAAAAAAACHI/YyuBoNYpXOQ/s800/google_app_engine_proxy.png)

A couple of points:

Taking
care of a VPS can be challenging, especially if you aren’t a Linux
user. However failing to do so can result in it being taken over by
malicious individuals. Consider getting a managed VPS. Also, three quick
security tips: use strong passwords. move the SSH server to a
non-standard port and configure your firewall to be as restrictive as
possible (deny everything, and open ports to only a limited number of IP
addresses).
Yes, this introduces a single point of failure
into your application, so plan accordingly (while your application is
small, this shouldn’t be a problem - as it grows you can get two VPS’s
at two different data centers for example for added reliability).
The
traffic between Google and your VPS can be encrypted using TLS (HTTPS).
The good news is that the URL fetcher service doesn’t check the
certificates, so you can use self-signed ones. The bad news is that the
URL fetcher doesn’t check the certificates, so a determined attacker can
relatively easily man-in-the-middle you (but it protects the data from
the casual sniffer).
Be aware that you need to budget for
double of the amount of traffic you estimate using at the VPS (because
it needs to first download it and then upload it back to Google). The
URL fetcher service does know how to use gzip compression, so if you are
downloading mainly text, you shouldn’t have a bandwidth problem.
PHP
might seem like an unusual choice of language, given how most GAE users
have experience in either Python or Java, but there are a lot of
tutorials out there on how to install it (and on modern Linux
distribution it can be done in under 10 minutes with the help of the
package manager) and it was the one I was most comfortable with as an
Apache module.

Without further ado, here are the relevant sourcecode snippets (which I hereby release into the public domain):

The PHP script:

 0) {
   curl_setopt($ch, CURLOPT_POST, true);
   $postfields = array();
   foreach ($_POST as $key => $val) {
       $postfields[] = urlencode($key) . '=' . urlencode($val);
   }
   curl_setopt($ch, CURLOPT_POSTFIELDS, implode('&', $postfields));
}

$headers = array();
function header_callback($ch, $header) {
   global $headers;
   // we add our own content encoding
   // also, the content length might vary because of this
   if (false === stripos($header, 'Content-Encoding: ')
       && false === stripos($header, 'Content-Length: ')
       && false === stripos($header, 'Transfer-Encoding: ')) {
       $headers[] = $header;
   }
   return strlen($header);
}

curl_setopt($ch, CURLOPT_HEADERFUNCTION, 'header_callback');

$output = curl_exec($ch);
if (FALSE === $output) {
   header("HTTP/1.0 500 Server Error");
   print curl_error($ch);
   exit;
}
curl_close($ch);

foreach ($headers as $header) {
   header($header);
}
print $output;

The “privatedata.php” file looks like this:

Two
separate files are being used to avoid submitting the password to the
source repository, while still keeping all the sourcecode open.

Now with the code in place, you can test it using curl:

curl --compressed -v -H 'X-Shared-Secret: 23tqhwfgj2qbherhjerj' 'http://127.0.0.1:1483/fetch_url.php?q=http%3A//identi.ca/api/help/test.json

As
you can see, a custom header is used for authentication. An other
security measure is to use a non-standard port. Limiting the requests to
the IPs of the Google datacenter from the firewall would be the ideal
solution, but given that this was the problem we are trying to solve in
the first place (the Google datacenters not having an officially
announced set of IP addresses), this doesn’t seem possible.

Finally, here is the Python code to use the script from withing an application hosted on the GAE:

from urllib import quote
from google.appengine.api import urlfetch

# ...

fetch_headers['X-Shared-Secret'] = '23tqhwfgj2qbherhjerj'
result = urlfetch.fetch(url='http://127.0.0.1:1483/fetch_url.php?q=%s' % quote(url), payload=data,
 method=urlfetch.POST if data else urlfetch.GET,
 headers=fetch_headers, deadline=timeout, follow_redirects=False)

A little more discussion about the code:

The
method of using a custom header for authorization was chosen, since the
forwarding of authentication data (ie. the “Authorization” header) was
needed (specifically this is what the Twitter API uses for verifying
identity)
Speaking of things the script forwards: it does
forward the user agent and any POST data if present. The forwarding of
other headers could be quite easily be added.
Passing
variables in a GET request is also supported (they would be
double-encoded, but that shouldn’t be a concern in but the most extreme
cases)
If we are talking about sensitive data, cURL (and the cURL extension for PHP) has the ability to fetch HTTPS content and to verify the certificates.

While
this method might look cumbersome, in practice I found it to work quite
well. Hopefully this information will help others facing the same
problem. A final note: if you have questions about the code or about
other aspects, post them in the comments. I will try to answer them as
fast as possible. I’m also considering launching a “proxy service” for
GAE apps to make this process much more simpler (abstracting away the
setup and administration of the VPS), so if you would be interested in
paying a couple of bucks for such a service, please contact me either
directly or trough the comments.

Helper for testing multi-threaded programs in Java

Attila-Mihaly Balazs — Sat, 27 Oct 2012 22:48:00 +0300

This post was originally published on the Transylvania JUG blog.

Testing multi-threaded code is hard. The main problem is that you
invoke your assertions either too soon (and they fail for no good
reason) or too late (in which case the test runs for a long time,
frustrating you). A possible solution is to declare an interface like
the following:

interface ActivityWatcher {
 void before();
 void after(); 
 void await(long time, TimeUnit timeUnit) throws InterruptedException, TimeoutException;
}

It is intended to be used as follows:

“before” is called before the asynchronous task is delegated to an
execution mechanism (threadpool, fork-join framework, etc) and it
increments an internal counter.
“after is called after the asynchronous task has completed and it decrements the counter.
“await” waits for the counter to become zero

The net result is that when the counter is zero, all your asynchronous tasks have executed and you can run your assertions. See the example code. A couple more considerations:

There should be a single ActivityWatcher per test (injected trough constructors or a dependency injection framework)
In production code you will use a dummy/noop implementation which removes any overhead.
This only works for situations where the asynchronous are kicked of
immediately. Ie. it doesn’t work for situations where we have
periodically executing tasks (like every 5 seconds) and we would want to
wait for the 7th tasks to be executed for example.

One thing the above code doesn’t do is collecting exceptions: if the
exceptions happen on different threads than the one executing the
testrunner, they will just die and the testrunner will happily report
that the tests passs. You can work around this in two ways:

use the default UncaughtExceptionHandler
to capture all exceptions and rethrow them in the testrunner if they
arrise (not so nice because it introduces global state – you can’t have
two such tests running in parallel for example)
Extend activity watcher and code calling activity watcher such that it has a “collect(Throwable)” method which gets called with the uncaught exceptions and “await” rethrows them.

Implementing this is left as an exercise to the reader :-).

GeekMeet talk about Google App Engine

Attila-Mihaly Balazs — Sat, 27 Oct 2012 22:45:00 +0300

The GAE presentation I’ve given for the 12th edition of Cluj Geek Meet can be found here (created using reveal.js).

You can find the source code here.

Lightning talk at Cluj.PM

Attila-Mihaly Balazs — Sun, 19 Aug 2012 15:22:00 +0300

The slides from my Cluj.PM lightning talk:

It was a stressful (but fun!) experience. Thanks to the organizers!

Running pep8 and pylint programatically

Attila-Mihaly Balazs — Sun, 19 Aug 2012 15:12:00 +0300

Having tools like pep8 and pylint are great, especially given the huge amount of dynamism involved in Python - which results in many opportunities to shooting yourself in the foot. Sometimes however you want to invoke these tools in more specialized ways, for example only on the files which changed since the last commit. Here is how you can do this from a python script and capture their output for later post-processing (maybe you want merge the output from both tools, or maybe you want to show only the lines which changed since the last commit, etc):

import pep8
try:
  sys.stdout = StringIO()
  pep8_checker = pep8.StyleGuide(config_file=config, format='pylint')
  pep8_checker.check_files(paths=[ ...path to files/dirs to check... ])
  output = sys.stdout.getvalue()
finally:
  sys.stdout = sys.__stdout__

from pylint.lint import Run
from pylint.reporters.text import ParseableTextReporter

reporter = ParseableTextReporter()
result = StringIO()
reporter.set_output(result)
Run(['--rcfile=pylint.config'] + [ ...files.., ], reporter=reporter, exit=False)
output = result.getvalue()

It is recommended that you use pylint/pep8 installed trough pip/easy_install rather than the Linux distribution repositories, since they are known to contain outdated software. You can check for this via code like the following:

if pkg_resources.get_distribution('pep8').parsed_version < parse_version('1.3.3'):
    logging.error('pep8 too old. At least version 1.3.3 is required')
    sys.exit(1)
if pkg_resources.get_distribution('pylint').parsed_version < parse_version('0.25.1'):
    logging.error('pylint too old. At least version 0.25.1 is required')
    sys.exit(1)

Finally, if you have to use an old version of pep8, the code needs to be modified to the following (however, this older version probably won’t be of much use and will most likely annoy you - you should really try to use an up-to-date version - for example you could isolate this version using virtualenv):

result = []
import pep8
pep8.message = lambda msg: result.append(msg)
pep8.process_options(own_code)
for code_dir in [ ...files or dirs... ]:
    pep8.input_dir(code_dir)

Clearing your Google App Engine datastore

Attila-Mihaly Balazs — Wed, 15 Aug 2012 10:08:00 +0300

Warning! This is a method to erase the data from your Google App Engine datastore. There is no way to recover your data after you go trough with this! Only use this if you’re absolutely certain!

If you have a GAE account used for experimentation, you might like to clean it up sometimes (erase the contents of the datastore and blobstore associated with the application). Doing this trough the admin interface can become very tedious, so here is an alternative method:

Start your Remote API shell

Use the following code to delete all datastore entities:

while True: keys=db.Query(keys_only=True).fetch(500); db.delete(keys); print "Deleted 500 entries, the last of which was %s" % keys[-1].to_path()

Use the following code to delete all blobstore entities:

from google.appengine.ext.blobstore import *
while True: list=BlobInfo.all().fetch(500); delete([b.key() for b in list]);  print "Deleted elements, the last of which was %s" % list[-1].filename

The above method is inspired by this stackoverflow answer, but has the advantage that it does the deletion in smaller steps, meaning that the risk of the entire transaction being aborted because of deadline exceeded or over quota errors is removed.

Final caveats:

This can be slow
This consumes your quota, so you might have to do it over several days or raise your quota
The code is written in a very non-pythonic way (multiple statements on one line) for the ease of copy-pasting

Relaxed JSON parsing

Attila-Mihaly Balazs — Tue, 27 Dec 2011 13:40:00 +0200

This blogpost was originally posted to the Transylvania JUG blog.

JSON is a good alternative when you need a lightweight format to specify structured data. But sometimes (for example when you want the user to specify JSON manually) you would like to relax the formalism required to specify “valid” JSON data. For example the following snippet is not valid as per the spec, although its intent is quite clear:

[{ foo: 'bar' }]

To make this standard compliant we would need to write it as:

[{ "foo": "bar" }]

We shouldn’t run out and blame the standard of course since it needs to balance many contradictory requirements (ambiguity of encoded data, ease of understanding, ease of writing parsers, etc). If you decide that you want to strike the balance differently (make the definition of valid data more relaxed) you can do this easily with the Jackson parser:

JsonParser parser = new JsonFactory()
    .createJsonParser("[{ foo: 'bar' }]")
        .enable(JsonParser.Feature.ALLOW_UNQUOTED_FIELD_NAMES)
        .enable(JsonParser.Feature.ALLOW_SINGLE_QUOTES);
JsonNode root = new ObjectMapper().readTree(parser);

assertEquals("bar", root.get(0).get("foo").asText());

If your tool of choice is gson, it is slightly more complicated but still doable. See the linked source code for a complete example.

JSON is a good tool for semi-structured data and using a relaxed parsing can make the programs you write easier to use.

Updating the root certificates for Java

Attila-Mihaly Balazs — Thu, 20 Oct 2011 15:36:00 +0300

One usually thinks of SSL in the context of HTTPS, but there are also other protocols which rely on it to provide security. See this link for a short overview of SSL – it only mentions HTTPS, but the same applies for IMAPS, FTPS, etc – SSL is independent of the wrapped protocol. You can have issues with your Java programs in where the party you are communicating with provider changes their certificate and the program rejects it as invalid. The exception is something like:

javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: 
    PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: 
    unable to find valid certification path to requested target

One cause of the problem can be that the server uses an SSL provider which is based on a root certificate that wasn’t included with the particular version of Java you are using (this is especially true for really old versions like Java 1.5). The issue can be solved by updating to the latest version, but it might be that this isn’t an option. Fortunately I found the following article: No more ‘unable to find valid certification path to requested target’

How to use it:

Compile the program javac InstallCert.java
Run it with the target host/port. For example in our case it would be: java InstallCert imap.mailprovider.com:993 (993 is the port for IMAPS)
navigate trough the menus and select which certificate to import
now you have a file called jssecacerts. You need to copy this to $JAVA_HOME/jre/lib/security/cacerts (back up the existing file first!)
Now the root certificate is imported (you can confirm this by rerunning InstallCert)

HTH

Vagrant and VirtualBox on Windows

Attila-Mihaly Balazs — Thu, 20 Oct 2011 15:27:00 +0300

Vagrant is a collection of scripts written in Ruby to manage VirtualBox images in a shared environment (like the QA boxes inside a company): install them, update them, etc. Unfortunately installing it under Windows is not as straight forward as one would want, so here are some useful tips:

Read the Windows setup page and the Windows x64 setup page (if you are on 64 bit)

If you are on a 64 bit Windows install:

Check out this post if your JRuby is using the 32 bit JVM on a x64 Windows setup
You need to use version 4.0 of VirtualBox (rather than the latest). You can get it from here

You need to use an older version of Vagrant:

jgem install jruby-openssl jruby-win32ole
jgem install --version '=0.7.8' vagrant

If the vagrant box download stops around 4G, check that you have a NTFS filesystem (rather than FAT) and deactivate any “network” scanning capabilities of installed security software (I had problems with NOD32 particularly)

HTH

Another friend blogging

Attila-Mihaly Balazs — Thu, 13 Oct 2011 18:20:00 +0300

Another friend ~~bit the dust~~ started blogging: Cleonte’s GitHub blog - bookmarks at the moment but looking forward to more involved posts :-).

Using Jython from Maven

Attila-Mihaly Balazs — Thu, 13 Oct 2011 18:16:00 +0300

This blogpost was originally posted to the Transylvania JUG blog.

On the surface it looks simple: just add the dependency and you can run the example code.

However what the jython artifact doesn’t get you are the standard python libraries like re. This means that as soon as you try to do something like the code below, it will error out:

PythonInterpreter interp = new PythonInterpreter();
try {
  interp.exec("import re");
} 
catch (PyException ex) {
  ex.printStackTrace();
}

The solution? Use the jython-standalone artifact which includes the standard libraries. An other advantage is that it has the latest release (2.5.2) while jython lags two minor revisions behind (2.5.0) in Maven Central. A possible downside is the larger size of the jar.

org.python
jython-standalone
2.5.2

How to post high-quality videos to Google Video

Attila-Mihaly Balazs — Tue, 11 Oct 2011 11:58:00 +0300

If you have Google Apps for Business, Google Video is still the preferred method of storing videos “in the cloud”: it is easier to embed into Google Docs and probably more importantly - it can be truly private (you can only access it if you are logged in with the correct account - as opposed to YouTube where you can “hide” the video, but still anyone with the link can access it).

To post a high quality video to Google Video you only have to do:

Upload the video
Wait

I kid you not :-). While initially the uploaded video is of poor quality, apparently it gets processed in the background and later on (in my experience: it takes around 30 minutes to process 1 hour of uploaded video) a high quality version will be available.

HTH

Integrating Maven with Ivy

Attila-Mihaly Balazs — Tue, 11 Oct 2011 11:18:00 +0300

This post was originally published on the Transylvania JUG blog.

The problem: you have some resources in an Ivy repository (and only there) which you would like to use in a project based on Maven. Possible solutions:

Migrate the repository to Maven (Nexus for example) since Ivy can easily use Maven-style repositories (so your Ivy clients can continue to use Ivy with some slight configuration changes and Maven clients will also work – also the push-to-repo process needs to be changed)
Try JFrog Artifactory since it reportedly can serve the same resources to both Ivy and Maven (disclaimer: I haven’t tried to use it actually and I don’t know if the Open Source version includes this feature or not)
or read on…

My goal for the solution (as complex as it may be) was:

It should be as simple and self-explanatory as possible
It should respect the DRY principle (Don’t Repeat Yourself)
It shouldn’t have other dependencies than Maven itself

The solution looks like the following (for the full source check out the code-repo):

Have two Maven profiles: ivy-dependencies activates when the dependencies have already been downloaded and ivy-resolve when there are yet to download. This is based on checking the directory where the dependencies are to be copied ultimately:

...
ivy-dependencies

false

${basedir}/ivy-lib


...
ivy-resolve

false

${basedir}/ivy-lib


...

Unfortunately there is a small repetition here, since Maven doesn’t seem to expand user-defined properties like ${ivy.target.lib.dir} in the profile activation section. The profiles also serve an other role: to avoid the consideration of the dependencies until they are actually resolved.

When the build is first run, it creates the target directory, writes the files needed for an Ivy build there (ivy.xml, ivysettings.xml and build.xml – for this example I’ve used some parts from corresponding files of the Red5 repo), runs the build and tries to clean up after itself. It also creates adependencies.txt file containing the chunck of text which needs to be added to the dependencies list. Finally, it bails out (fails) instructing the user to run the command again.

On the second (third, fourth, etc) run the dependencies will already be present, so the resolution process won’t be run repeteadly. This approach was chosen instead of running the resolution at every build because – even though the resolution process is quick quick – it can take tens seconds in some more complicated cases and I didn’t want to slow the build down.

And, Ivy, the Apache BSF framework, etc are fetched from the Maven central repository, so they need not be preinstalled for build to complete successfully.

A couple of words about choosing ${ivy.target.lib.dir}: if you choose it inside your Maven tree (like it was chose in the example), you will receive warnings from Maven that this might not be supported in the future. Also, be sure to add the directory to the ignore mechanism of your VCS (.gitignore, .hgignore, .cvsignore, svn:ignore, etc), as to avoid accidentally committing the libraries to VCS.

If you need to add a new (Ivy) dependency to the project, the steps are as follows:

Delete the current ${ivy.target.lib.dir} directory
Update the part of your pom.xml which writes out the ivy.xml file to include the new dependency
Run a build and watch the new dependency being resolved
Update the dependencies section of the ivy-dependencies profile to include the new dependency (possibly copying from dependencies.txt)

One drawback of this method is the fact that advanced functionalities of systems based on Maven will not work with these dependencies (for example dependency analisys / graphing plugins, automated downloading of sources / javadocs, etc). A possible workaround (and a good idea in general) is to use this method for the minimal subset – just the jars which can’t be found in Maven central. All the rests (even if they are actually dependencies of the code fetched from Ivy) should be declared as a normal dependency, to be fetched from the Maven repository.

Finally I would like to say that this endeavour once again showed me how flexible both Maven and Ivy/Ant can be and clarified many cornercases (like how we escape ]] inside of CDATA – we split it in two). And it can also be further tweaked (for example: adding a clean target to the ivy-resolve profile, so you can remove the directory with mvn clean -P ivy-resolve or re-jar-ing all the downloaded jars into a single one for example like this, thus avoiding the need to modify the pom file every time the list of Ivy dependencies gets changed – then again signed JARs can’t be re-jarred so it is not an universal solution either).

Upgrading the Options (GlobeTrotter) GI515m

Attila-Mihaly Balazs — Sat, 08 Oct 2011 15:55:00 +0300

Recently I needed to install an Options (GlobeTrotter) GI515m 3G USB modem on a machine which previously used an older version of the modem (the iCON 225). This seems a pretty common scenario (an existing user getting an update), however the process seems less-than-straight forward:

Get a second computer with the same operating system version which didn’t have a 3G modem installed (for example if your target system is running Windows 7 64 bit you need a second system with Windows 7 64 bit - different skews like Home vs Ultimate are ok, but the version and “bitness” must coincide - you could also try using a virtual machine for the second machine which supports USB forwarding like VirtualBox or VMWare)
Plug in the modem in the second machine. First it will recognize it as an USB stick / CD-ROM. Copy all the files from it to a separate folder (you should see files like “setup.exe“).
Let the setup complete. Now copy the installed drivers to the same place you’ve saved setup file. Under Windows 7 you would find them in the location C:\Windows\System32\DriverStore\FileRepository\ in several folders starting with “gth” (like gthsubus_64.inf_amd64_neutral_4810563f34b37ef5), but here is the generic way to identify the folder:
1. Start Device Manager
2. Look for one of the devices associated with the modem (you will find actually several, like GlobeTrotter GI515M - Modem Interface, Network Interface and so on)
3. Properties -> Driver -> Driver Details. Note the name of the driver for which the provider is Option (for example gtuhsser.sys)
4. Now search your Windows folder for files ending in .inf which contain the name of driver from the previous step. This will point you to the right folders
On the first computer (the one you actually want to install the modem on) remove all previous versions of the software using the Add-Remove Programs facility (you will see two-three entries but they can be easily identified by the same orange icon). Restart the computer for good measure.
Copy over the setup program and the drivers from the second computer. Plug in the modem to the first computer, install the application (using the setup file captured on the second computer). Go into the device manager and look for “Unknown device”s (you should see four of them). Use the drivers captured on the second computer to resolve these issues.
Unplug and replug the modem - it now should work!

A couple more talking points:

don’t use “driver manager” type software - they very rarely (read: never) seem to work
a symptom that you’ve hit this problem is when the management interface (dialer / “Internet Everywhere”) for the modem starts but it gets stuck in the “Initializing” phase when you connect the modem and consumes CPU (from what I’ve seen with a debugger it seems to be looking for the installed device in a loop)
the modem seems to be prone to overheating if the signal-strength is low (around two bars) and in this case it shuts down after \~10 minutes (I assume that this is some kind of thermal protection). You can check if this is the case by putting your hand on the bottom side of the modem. I couldn’t find and solution for this, other than looking for a spot which has better signal. Using the modem in EDGE rather than 3G mode also seems to do the trick, but it has lower speeds and I don’t know of any reliable method to make the modem use EDGE if 3G is also available.

Getting the most out of your audio recording with Audacity

Attila-Mihaly Balazs — Sat, 08 Oct 2011 13:45:00 +0300

This article aims to show you some simple techniques to improve the quality of your voice recording quickly and cheaply (for free actually). But first things first:

The best audio is the one you don’t have to improve. Some simple steps you can perform in advance to maximize quality:

Use quality equipment. Here are some articles about the equipment great-sounding podcasters use. You don’t have to spend a lot of money, but definitely stay away from the built-in laptop microphone
Eliminate ambient noise as much as possible (close windows, draw the blinds, stop other electronic equipment in the room, etc)
Record each person on a separate channel - if possible on a computer local to them (avoid recording trough Skype, GoToMeeting or other VoIP solutions)
Try keeping the recording volume for each microphone at the optimal level – not too low, but also avoiding clipping

After you have the audio recording there is still a lot you can do, but it is preferable to start out with the best source material. For the example below I’ll be using the raw recordings from a recent SE Radio podcast:

The situation with this recording is as follows:

There are separate audio tracks for the interviewer and interviewee (good)
There is background noise on the tracks (easily correctable)
Both persons were picked up by both microphones (correctable)
The interviewer has some clipping (partially correctable – luckily it’s not the interviewee who has clipping)

The steps to improve the quality of this recording are as follows:

First, install the Noise Gate plugin for Audacity, since it requires program restart (under Windows you have to copy the downloaded noisegate.ny to C:\Program Files (x86)\Audacity 1.3 Beta (Unicode)\Plug-Ins or to a similar location, under Linux you have to place it in /usr/share/audacity). After copying the file you have to close and restart Audacity. To verify that the plugin was properly installed check in the Effect menu – you should see an entry title “Noise gate”.

Now that we have Audacity all set up and the plugin installed, first split the stereo track into mono tracks, since they don’t actually represent left-right channels but rather two speakers which will be mixed together at the end. For this click on the arrow after the filename in the track and select “Split Stereo to Mono”. Sidenote: some people will prefer to mix different speakers in podcasts with different panning (that is to the left or to the right). I would advise against this: it is distracting if you are doing something else while listening to the podcast (like walking / jogging / riding a bike / etc). It can also backfire if for some reason the listening device is missing one of the channels (the “damaged headphone” scenario).

The first thing will be to remove the constant background noise (like AC hum for example). To do this zoom in (Ctrl + 1) and look for low volume zones. Select those zones and go to Effects –> Noise Removal –> Get Noise Profile. Now select a zone where the noise is mixed with speech and test out the settings (Effect –> Noise Removal –> Ok). After the test you can use Undo (Ctrl + Z) to roll back the changes. You should watch for the noise being removed but also the natural sound of the voice being preserved (too aggressive of a noise removal can lead to a “robot voice” effect). If you are satisfied, you can go ahead and apply it to the entire track. Also, since the noise source might change during the recording, you should at least do a quick scroll to check for other low-volume zones which can be a sign of noise. If you find noise from other sources, you can use the same steps to remove it.

Now that you have removed the noise, the next step would be to remove the voices from the channels they don’t belong to. This is where we’ll be using the Noise Gate plugin: since there is a considerable level difference between the wanted audio and the unwanted audio on each channel, we can just declare everything below a certain volume “noise” and use the plugin to silence it. A couple of tips:

This needs to be done separately for each channel, since the cutoff volume will be different
You can use the “Analyse Noise Level” function of the plugin to gauge the approximate level of the cutoff volume – this will only give you an estimate and you will have to play around with the settings a little bit to find the optimal volume
Use a “Level reduction” of –100 dB to completely filter out the sound and an “Attack/Decay” of 1000 milliseconds to avoid false positives
As with all the steps, you can experiment on a smaller portion of the audio file (since it is much quicker) to fine tune the settings by repeatedly applying the effect with different parameters and undoing (Ctrl+Z) the result after evaluation. When the parameters seem right, just select the entire track and press Ctrl+R (Repeat last effect)

After we’ve finished with both tracks, we have a better situation:

Now we will fix the clipping as much as possible (a perfect fix isn’t possible since clipping means that information got lost and all the plugins can do is to “guess” what the information might have looked like). First we reduce the aplification of the second track (the one which contains the clipping) by 10 dB as the Clip Fix plugin suggests (Effect –> Aplify –> –10 dB) after which we use the Clip Fix plugin. Unfortunately this plugin runs very slowly if we would to apply it to the entire track at once. Fortunately we have a reasonable workaround: select portions of the track and apply the plugin to them individually. After the first application you can use the “Repeat last effect” shortcut (Ctrl+R) to speed up the operation. Sidenote: it is a good habit to use the “Find Zero Crossing” function whenever you do a selection (the shortcut is Z – so whenever you select a portion, just press Z afterwards). This eliminates some weird artifacts when cutting / pasting / silencing part of the audio and it might even help when applying different effects. The fixed audio looks like this:

Now, that all the cleanup steps have been performed, there is one last step which is as important as the cleanup: maximizing the audible volume without introducing clipping. This is very important because all devices can reduce volume but few of them can increase it (some exceptions being: the Linux audio stack and VLC). The easiest way to do this is by using the Levelator (note: while the Levelator is free – as in beer – and does not restrict what you can do with the output, it is not free as in freedom if this is a consideration for you).

To do this, export the audio to WAV (make sure that all tracks are unmuted during export) and run the Levelator on it. The end result will look like the following:

Of course the Levelator isn’t magic pixie dust either, so here are a couple of things to check after it has been run:

Did it amplify some residual noise which wasn’t available in the initial audio? (if so, you should remove it using the Noise Removal plugin)
Did it miss segments? (it is rare, but it happens – those segments need to be amplified manually)
It results in “weird” sounding audio if the recording has been preprocessed by a dynamic compressor – for example GoToMeeting has an option to improve sound quality which uses dynamic compression and thus makes the recording unsuitable for the use with Levelator

That’s it for this rather long article. Don’t be discouraged by the length of the article: after going over the steps a couple of times, it shouldn’t take longer than 15 minutes to process a 2 hour interview (excluding the cutting / pasting / moving parts around) and you will gain listeners because of the higher production value.

A final note on the output formats: while during processing you should always use lossless formats, the final output format I recommend is: MP3 at 64 kbps CBR, Joint Stereo, 22050 MHz sampling rate. I found that this is the best balance between quality, file size and compatibility with the most playback devices out there.

Of maps and men

Attila-Mihaly Balazs — Wed, 21 Sep 2011 17:53:00 +0300

A very cool visualization of the imigration / emigration data:

[![](http://2.bp.blogspot.com/-Dwdqbim7Fm4/Tnn4P5DPHZI/AAAAAAAAEU0/iLOqw-j2zU8/s320/migrationsmap.png)](http://migrationsmap.net/#/JPN/departures)

To remember: the relative sizes of countries / continents on most maps is not representative of the true ratios, because most map projections were not meant for that. If you want to play around with different projections, here is a nice page from Wolfram (unfortunately you have to install a \~100MB plugin to get it to work). If you need the raw data, just go to Wikipedia.

Finally, here is a good essay from Asimov (from 1989) about the scientific process: The Relativity of Wrong - small nitpick: it would have been even better if it had used the metric system or at least it didn’t switch from miles to inches in he middle of the essay.

Informative videos about copyright and remixing issues

Attila-Mihaly Balazs — Wed, 21 Sep 2011 17:02:00 +0300

Walking on Eggshells: Documentary about Remix Culture - via the comixtalk blog:

Internet is Freedom - a speech given by Lawrence Lessig at the Italian Parlament - via the Security4All blog:

PBS: Copyright Criminals - while the video is not online, you can watch the trailer and listen to some remixes inspired by it which are under a CC license.

And just as a bonus: a cynical video about filtering the Internet - via the IT Law in Ireland blog:

Update:

The History of Copyright Law - via the laughing squid:

TED Johanna Blakley: Lessons from fashion’s free culture

Update: it seems that I already posted the last video - admittedly I’m becoming senile with the advance of the years :-)

Recording test performance with Jenkins

Attila-Mihaly Balazs — Tue, 20 Sep 2011 19:16:00 +0300

In many (most?) systems performance is an important non-functional requirement. And even if you attained the required performance, it is useful to keep an eye on it to detect if a codechange involuntarily deteriorates it. Enter the Performance plugin for Jenkins. Using it you can record the performance (as in: speed of execution) of your test runs and set alter thresholds which cause the build to fail. Also it can generate graphs like the one below:

To do this:

Have Jenkins installed
Intstall the Performance plugin (or upgrade to the latest version, since there was a bug in earlier versions which prevented the parsing of the JUnit reports)
For your build check “Publish Performance test result report” and add locations where the reports should be collected from.
That’s it! Future builds will collect the performance data and you can access it using the “Performance Trend” link (at the job level) or the “Performance Report” link (at the build level)

More details / caveats:

The paths are defined as ANT file expressions (that is you can use “**” to specify an arbitrary level of directories, for example: target/surefire-reports/**/TEST*.xml)
JUnit performance is grouped at the test-class level, thus it probably makes sense create separate project / module which group the performance test cases.
Benchmarking is hard and JUnit doesn’t give you any provisions to do warmup or to repeat the tests multiple times. To make your test as relevant as possible you should do this manually (warmup code can be placed in the @Before method for example). A properly set up JMeter task accounts for this already.
TestNG tests can also be parsed as long as the test run is set to produce a JUnit compatible report.
Slightly off-topic: to integrate a JMeter run into your maven build, you can use the AntRunplugin:
```
maven-antrun-plugin
1.6

test

run
```

Article originally posted to the Transylvania JUG blog.

Audio quality redux

Attila-Mihaly Balazs — Tue, 20 Sep 2011 15:20:00 +0300

Yet an other example for how simple steps can improve the audio quality considerably. The clip below is taken from this blogpost (which I originally found trough Hacker News). You can find the processed version here, or use the controls below to do a quick A/B comparison of the two. The processing was very simple (1. noise removal and 2. running trough the Levelator) and quick.

Position:

Volume:

Crossfade (Original - New):

PS: For people reading the post trough an RSS reader: you probably need to click trough to the site to see the comparison in action, since most (all?) RSS readers filter out javascript for security reasons.

PS: If you are interested in the simple script which was use to interact with the two youtube players, you can find it in my code repository.

Power Line Humm Removal With Audacity

Attila-Mihaly Balazs — Sun, 18 Sep 2011 16:45:00 +0300

As a response to George Starcher’s Removing Power Line Hum from Audio with GarageBand I would like to post a quick tutorial on how to do the same with Audacity:

Protein Shakes site review

Attila-Mihaly Balazs — Fri, 16 Sep 2011 20:33:00 +0300

[![](http://www.strongernutrition.com/productimages/400216_500_500.jpg)](http://www.strongernutrition.com/productimages/400216_500_500.jpg)

This is something new for me: protein shakes. Medically I can’t offer advice about it (I’m weary about using foreign substances not recommended by a specialist), but the site certainly is has some positive signs:

The badges at the bottom are clickable and they go to the respective sites which certify the site
The domain is registered since 2006 and they are paypal verified since 2007
They have an active Facebook and Twitter account
They use paypal / google checkout for payment which reduces your risk considerably

They also have also some less positive signs:

There is no physical contact address
The registration details are hidden by proxy registration
The phone-number is a generic one (I would have liked one which coincides with the physical location)

All in all I would recommend this site for small purchases from inside the USA (they don’t deliver internationally). Also, I would consult a doctor (or multiple doctors) about the possible effect of the substances. In my view (and I’m no doctor) this is more serious than the homeopathic or “natural” substances and it should be handled with care. Then again, I’m also against medicine / medical devices (like glasses) being sold in places without expert supervision (like supermarkets).

Full disclosure: this is a paid review from ReviewMe. Under the terms of the understanding I was not obligated to skew my viewpoint in any way (ie. only post positive facts).

Running JRuby on 64 bit Windows

Attila-Mihaly Balazs — Tue, 13 Sep 2011 19:33:00 +0300

Usually it is as simple as: download, install, run. You can run into problems however if you have both the 32 bit and 64 bit JVMs installed (which is quite often) because it will try to use the 32 bit JVM. You can check which JVM is being used from the command line:

jruby --version
jruby 1.6.3 (ruby-1.8.7-p330) (2011-07-07 965162f) (Java HotSpot(TM) 64-Bit Server VM 1.7.0) [Windows 7-amd64-java] # 64 bit
jruby 1.6.3 (ruby-1.8.7-p330) (2011-07-07 965162f) (Java HotSpot(TM) Client VM 1.6.0_26) [Windows 7-x86-java] # 32 bit

To work around this issue, specify the JVM to use in your jruby.bat (or other batch files installed by gems like vagrant.bat) explicitly. Example jruby.bat:

@ECHO OFF
java -Djruby.home=C:\jruby-1.6.3 -jar  -jar "C:\jruby-1.6.3\lib\jruby.jar" %1 %2 %3 %4 %5 %6 %7 %8 %9

Example vagrant.bat

@ECHO OFF
java -Djruby.home=C:\jruby-1.6.3 -jar "C:\jruby-1.6.3\lib\jruby.jar" "C:/jruby-1.6.3/bin/vagrant" %1 %2 %3 %4 %5 %6 %7 %8 %9

Using less with syntax highlight

Attila-Mihaly Balazs — Tue, 13 Sep 2011 19:14:00 +0300

You can use vim as your pager and obtain two benefits: syntax highlight and access to all the advanced commands (like search). You can do this under ubuntu by adding the following line to your ~/.bashrc:

alias less='/usr/share/vim/vimcurrent/macros/less.sh'

Note:

You have to have vim installed (which doesn’t come by default, but it is as simple as sudo apt-get install vim-nox)
It supports viewing directly bz2 and gz archives as well as pipe input from stdin (but in that case it fails sometime to highlight)
Edit commands (like dd) are disabled, so you can’t accidentally modify the file you are viewing

Link love

Attila-Mihaly Balazs — Fri, 09 Sep 2011 18:23:00 +0300

Here are a couple of close friends’ blogs. They are just starting out writing, but hopefully giving them some link love will encourage them to write even more great content. Without further ado, in no particular order:

100 years of style

Attila-Mihaly Balazs — Tue, 06 Sep 2011 13:15:00 +0300

A very entertaining video in the style of Evolution of Dance:

Hattip to refresh.ro.

Quick’n’dirty Mediawiki file crawler

Attila-Mihaly Balazs — Tue, 06 Sep 2011 12:36:00 +0300

URL='http://10.0.0.1' MIME='image/jpeg'   
 bash -c 'wget -q -O - "$URL/wiki/index.php?title=Special:MIMESearch&mime;=$MIME&limit;=500&offset;=0"   
 | grep -Po "\/wiki\/images[^\"]+"   
 | xargs -n1 -I {} wget "$URL{}"'

What it does: it uses the “MIME search” functionality on the wiki to locate files of a certain mime type and then xargs+wget each of them.

Limitations:

A maximum of 500 files are downloaded
Downloads are not parallelized, thus slower than they could be

Creating a non-MAC bound CentOS 6 machine

Attila-Mihaly Balazs — Mon, 05 Sep 2011 10:04:00 +0300

I was building VMs to be deployed with Vagrant / Virtualbox for our QAs and discovered that on new instantiations of the machine the networking interface wasn’t coming up. The problem was that Virtualbox was assigning a random MAC address to the NIC (and rightly so, to avoid conflicts). I used the following steps to solve this:

Remove the HWADDR line from /etc/sysconfig/network-scripts/ifcfg/eth0
Delete the file /etc/udev/rules.d/70-persistent-net.rules (hat tip)

These two steps are specific to CentOS 6 (on 5.x the first step is sufficient). Also, the second if is recreated at the next boot, thus after rm-ing it, you should shut down the machine and package it (not start it again, or if you do, you should remove the file again).

Levant Digital Marketing review

Attila-Mihaly Balazs — Thu, 01 Sep 2011 16:15:00 +0300

Levant Digital Marketing is a company which does Search Engine Optimization in the Middle East. They seem to be a very new company (the domain was registered in January of 2011). They are part of “JHG Holding”, but this itself is also only from 2009 (and their site contains minimal content). I didn’t manage to find their headquarters on Google Maps either (but this might just be an issue with Google Maps in foreign countries). Their phone number is indeed from Lebanon (as their physical address is). Their facebook page is non-existent at the moment (was it deleted?) and the twitter account is completely empty.

All in all, while I couln’t find anything explicitly negative about them, it is more the case that I couldn’t find anything concrete about them :-). As far as advice goes, when starting out you are better off with some simple steps, and when you grow you might consider looking into a SEO consultancy, but take care to find a reputable one rather than a cheap one or one which promises you the moon.

Full disclosure: this is a paid review from ReviewMe. Under the terms of the understanding I was not obligated to skew my viewpoint in any way (ie. only post positive facts).

Paving site review

Attila-Mihaly Balazs — Wed, 03 Aug 2011 22:16:00 +0300

This is a fairly trustworthy site to buy paving slabs and other construction materials. It checks out on on all sources I usually use: domain registration, physical address, reputation sites and a quick search for complains. So go ahead and look around, but remember that for larger purchases you should consult an expert. They also have a fairly standard return policy, so if you do find out that you don’t need a certain item, you might still be able to salvage some money.

External pavement is tricky in general, so you really need to consult somebody who done it before. It requires some kind of foundation (mostly compacted sand) and manual work to lay each piece. When finished it can look very nice, but it can also be very adversarial to people / children who happen to fall on them. I would recommend grass, asphalt (possibly a small strip of it for the tires if we are talking about a garage) and only after that paving with slabs. Good luck with the construction!

Full disclosure: this is a paid review from ReviewMe. Under the terms of the understanding I was not obligated to skew my viewpoint in any way (ie. only post positive facts).

IVA site review

Attila-Mihaly Balazs — Wed, 03 Aug 2011 21:50:00 +0300

Reviewing this site pose a conundrum to me (other than how to write the word conundrum): on the one side they seem to be a legitimate site for IVA advice, with a long standing domain registration, a physical address and even an entry at the Office of Fair trading (which seem to be the equivalent of the Better Business Bureau). On the other hand there are other sites with the same profile - debt relief - which refer the same physical address but are hosted elsewhere (in Germany for example).

I can’t draw a definite conclusion about this. I’m not from the UK and have zero familiarity with the UK law. Probably the only advice I can give at this moment is to try asking your close family or your trusted friends. I imagine that it is painful to do so, but this is the only source I can think of where you can get money without the exorbitant rates asked by payday lenders. All that I can do is hope that you prevail, since I have a deep belief that as long as you have determination, you will find your place.

Full disclosure: this is a paid review from ReviewMe. Under the terms of the understanding I was not obligated to skew my viewpoint in any way (ie. only post positive facts).

Setting up git-daemon under Ubuntu

Attila-Mihaly Balazs — Wed, 01 Jun 2011 20:19:00 +0300

The scenario is the following: inside a (somewhat) trusted LAN you would like to set up git-daemon so that your coworkers can access your repositories. This solution is not appropriate in cases where you want to share with random people on the interwebs. This short description is based loosely on this blogpost and it was updated to contain more details and tested with Ubuntu 11.04.

install the git-daemon-runit package: sudo apt-get install git-daemon-runit
decide where you would like to keep your git repositories - it can be your home folder, if it’s not encrypted (if it’s encrypted it won’t work because it only gets decrypted once you log in, so the git repositories won’t be available unless you log in). Lets say that you’ve decided it to be /var/git. Create it:
sudo mkdir /var/git sudo chown $USER /var/git
Now edit the file /etc/sv/git-daemon/run and make it like the following (bold marks the spots which were changed):
#!/bin/sh exec 2>&1 echo 'git-daemon starting.' exec chpst -ugitdaemon \ "$(git --exec-path)"/git-daemon --verbose --export-all --base-path=/var/git /var/git
Restart the service:
sudo sv restart git-daemon
Enable it from the firewall:
sudo ufw allow 9418/tcp

That’s it. Now every subdirectory from /var/git which “looks like” a git repo (has a .git subdirectory) will be available over the git protocol. Alternatively, you can remove the “—export-all” option and create a “git-daemon-export-ok” file in each subdirectory you would like to export: touch /var/git/core/git-daemon-export-ok

You can symlink the directory to your home folder for your convenience:
ln -s /var/git ~/projects/git

Adding tab completition to Maven3 under Ubuntu

Attila-Mihaly Balazs — Wed, 01 Jun 2011 20:10:00 +0300

Maven 3 was released recently (depending on your definition of recent), but is not yet packaged for Ubuntu. This is generally not a problem, since the installation instructions are easy to follow (alternatively here are the installation instructions from the Sonatype maven book), but you don’t get tab completion in your terminal, which is quite a bummer, since I don’t know how to write correctly without a spellchecker.

Fortunately the steps to add it are simple:

Download an older Maven2 package
Extract from it the /etc/bash_completion.d/maven2 file (take care not to install the package by mistake)
Put the extracted file into /etc/bash_completion.d/maven3
Restart your terminal

These steps should also work with other Linux distributions if they have bash-completion installed.

This is a cross-post from the Transylvania-JUG blog.

adulttoys.com review

Attila-Mihaly Balazs — Sat, 21 May 2011 09:53:00 +0300

Sex sells, but in this case it is the reverse: money sold me to review an adult sex toys website. Here we go:

I didn’t use any sextoys myself to-date (although I would be open to try some of the lighter versions), but I can see how online shopping and discreet delivery can be a big advantage. A word of caution: always consider the medical implications and don’t force yourself into un-natural situations which could end up hurting you.

Getting back to the store at hand:

It only delivers inside of the USA, so it is a no-go for me (but there are plenty of local stores here, so I’m not complaining)
Their site interface is pretty well made and usable, but there are bugs: for example when added one product to the cart and tried to check out, it appeared twice in the cart
One big plus: their entire site is available trough HTTPS. Even though they default to HTTP, this is still great.
Unfortunately their contact details are a little sketchy (and I mean it in the literal sense: there is very little information): they don’t give a physical address, just a PO box from San Diego. They are also hosted by a San Diego company, which makes it very probable that they are based there, but again, there is no clear indication of this. Their contact form doesn’t give an email address or a phone number. They are hosted on a dedicated server (good) but with IIS 6.0 (not so good). They are doing their own credit-card processing from what I’ve seen (I didn’t go trough with an actual order) and I would be weary entrusting my credit-card to a company which is not specialized in this domain (the CC processing domain that is).
The site has no negatives on security rating sites like SiteAdvisor, WOT or Norton SafeWeb.

All in all, I would do only small purchases on this site, at least initially, from a disposable credit-card (PayPal offers these for example)

Full disclosure: this is a paid review from ReviewMe. Under the terms of the understanding I was not obligated to skew my viewpoint in any way (ie. only post positive facts).

blackjackpractice.org review

Attila-Mihaly Balazs — Fri, 15 Apr 2011 20:53:00 +0300

I was hired to write a review about blackjackpractice.org which purportedly blackjack practice, however I wasn’t able to verify this, since the site is down (currently it is showing a default directory listing from Apache, earlier today it was showing an empty page). There is very little know about this by the usual sources, and there doesn’t seem to be anything interesting on the same server (it is hard to tell if the same person owns all the domain on the server due to the “privacy protected” registration).

Now back to the idea to blackjack practice: can you really train yourself? Probably, to some extent (hey, even Kent Beck - yes, that Kent Back - has a poker training website). With regular exercise you can memorize basic strategy table (this will help you with any game, regardless if its with a live dealer or with a machine - supposing that the machine is playing fair) and then you can move to more advanced techniques like card counting or shuffle tracking. I doubt however that you can achieve a level where this would be a profitable endeavor.

Full disclosure: this is a paid review from ReviewMe. Under the
terms of the understanding I was not obligated to skew my viewpoint in
any way (ie. only post positive facts).

Booting the Linux Kernel from Grub2

Attila-Mihaly Balazs — Tue, 12 Apr 2011 18:59:00 +0300

Recently a good friend of mine managed to uninstall all the kernels from his Ubuntu machine (what can I say - Monday morning and no coffee is a deadly combination). Luckily he had the install CD on hand so we did the following:

Boot from the CD (we had Internet connection)
Mount the Linux partition and chroot into it
Reinstall the kernel with aptitude
Reboot and go into Grub2 command mode

Now do the following (commands need to be adjusted to match your partition - also, tab completion works, so you don’t have to guess)

insmod part_msdos
insmod ext2
set root=(hd0,3)
linux /boot/vmlinuz-2.6.32.38-generic root=/dev/sda3 ro
initrd /boot/initrd.img-2.6.38-6-686
boot

It seems that most of the examples on the ‘net are for Grub 1 and little is out there for Grub 2. I found the following three: How to use Grub2 to boot Linux manually, The Grub 2 Guide, GRUB 2 bootloader - Full tutorial. Also, I didn’t perform steps 4-5 because he just reinstalled Ubuntu (it was a fresh install anyway), but I tried it out separately on my laptop and it works.

HTH

The wrong time to update software…

Attila-Mihaly Balazs — Mon, 11 Apr 2011 16:41:00 +0300

is when the user is the busiest, for example when s/he just started your application. See for example the screenshot below with Adobe Air (click trough to see it in its full beauty).

[![](http://3.bp.blogspot.com/-r_lf3uTwZgY/TaMCg6ba3pI/AAAAAAAADWA/d2bvVKLpVQE/s320/adobe-air-update.png)](http://3.bp.blogspot.com/-r_lf3uTwZgY/TaMCg6ba3pI/AAAAAAAADWA/d2bvVKLpVQE/s1600/adobe-air-update.png)

The mistakes it makes:

It tries to do the update when I’m trying to start Grooveshark (it interferes with my intention)
It consumes 100% of a core by polling for the presence of running applications (I suppose), effectively obliging me to do the update. This is combined with frequent releases (which otherwise would be a good thing) for maximum annoyance.
Although you can’t see it in the screenshot, the updater has (had?) a bug when it asks for your sudo password: if you misstype it at first, then it asks for the root password (which doesn’t exists under Ubuntu by default) and then it just gets into some weird state until the next update is released.

To sum it up: You should download and install the updates in the background (in a separate, versioned directory, always keeping just the two most recent versions). Users shouldn’t be bothered with this, especially when they are trying to get work done!

Recovering encrypted home directory under Ubuntu

Attila-Mihaly Balazs — Sun, 10 Apr 2011 22:24:00 +0300

While the home-folder encryption in Ubuntu is far from a perfect solution (there is considerable data leakage from the swap file and the temp directory - for example once I’ve observed the flash videos from Chromium ~~porn~~ private browsing mode being present in the /tmp directory), it is a partial solution nevertheless and very easy to set up during installation. However what can you do if you need to recover the data because you fubard your system?

Credit where credit is due: this guide is taken mostly from the Ubuntu wiki page. Also, this is not an easy “one-click” process. You should proceed carefully, especially if you don’t have much experience with the command line.

Start Ubuntu (from a separate install, from the LiveCD, etc) and mount the source filesystem (this is usually as simple as going to the Places menu and selecting the partition)
Start a terminal (Alt+F2 -> gnome-terminal) and navigate to the partitions home directory. Usually this will look like the following:
```
cd /media/9e6325c9-1140-44b7-9d8e-614599b27e05/home/
```
Now navigate to the users ecryptfs directory (things to note: it is ecryptfs not encryptfs and your username does not coincide with your full name - the one you click on when you log in)
```
cd .ecryptfs/username
```
The next step is to recovery your “mount password” which is different from the password you use to log in (when it asks you, type in the login password used for this account - for which you are trying to recover the data). Take note of the returned password (you can copy it by selecting it and pressing Shift+Ctrl+C if you are using the Gnome Terminal)
```
ecryptfs-unwrap-passphrase .ecryptfs/wrapped-passphrase
```
Now create a directory where you would like to mount the decrypted home directory:
```
sudo mkdir /media/decrypted
```

Execute the following and type in (or better - copy-paste) the mount password you’ve recovered earlier

sudo ecryptfs-add-passphrase --fnek

It will return something like the following. Take note of the second key (auth tok):

Inserted auth tok with sig [9986ad986f986af7] into the user session keyring 
Inserted auth tok with sig [76a9f69af69a86fa] into the user session keyring

Now you are ready to mount the directry:

sudo mount -t ecryptfs /media/9e6325c9-1140-44b7-9d8e-614599b27e05/home/.ecryptfs/username/.Private /media/decrypted
 Passphrase:  # mount passphrase
 Selection: aes
 Selection: 16
 Enable plaintext passthrough: n 
 Enable filename encryption: y # this is not the default!
 Filename Encryption Key (FNEK) Signature: # the second key (auth tok) noted

You will probably get a warning about this key not being seen before (you can type yes) and asking if it should be added to your key cache (you should type no, since you won’t be using it again probably).

That’s it, now (assuming everything went right) you can access your decrypted folder in /media/decrypted. The biggest gotcha is that home/username/.Private is in fact a symlink, which - if you have an other partition mounted - will point you to the wrong directory, so you should use the home/.ecryptfs/username directory directly.

HTH

scentsy review take two

Attila-Mihaly Balazs — Sun, 10 Apr 2011 22:19:00 +0300

I’ve already written about Scentsy Products, so I will try not to repeat myself that much (other than reiterating that you should really think before investing in a referral system) and will focus on their special product:

Piece by Piece Full-Size Scentsy Warmer – this a usual warmer (usual for Scentsy that is – it uses a lightbulb to provide the heat, thus avoiding the open flame and smoke) with a puzzle-piece decoration. What makes this item (more) special is the fact that parts of the revenue from it go to Autism Speaks. While currently this is the only one in the Charitable Cause Warmers product line, hopefully there will be more in the future, allowing you to get something for both your body and your soul.

An other item I didn’t talk about in the last article is the gift certificate: if you consider appropriate, you could give a 25 USD Gift Certificate to the person. There are also replacement parts and individual warmer parts if your warmer breaks but you don’t want to buy a completely new one. Also, the light bulbs in the Scentsy products are standard ones (compared to something like a Philips wake-up light) so you can buy a replacement in almost any store, as long as you watch for the socket size and the wattage.

Full disclosure: this is a paid review from ReviewMe. Under the terms of the understanding I was not obligated to skew my viewpoint in any way (ie. only post positive facts).

rubberecycle.com review

Attila-Mihaly Balazs — Sun, 10 Apr 2011 11:17:00 +0300

There isn’t much I can say about this company. They seem very legitimate by all indications (domain name registered more than 10 years ago, with the physical address of the company, no complaints on the web, etc). Their goal seems also very laudable: creating playground surfacing out of recycled tire rubber. While I don’t have enough information to ascertain if this is truly more eco-friendly than getting rid of the tires in other ways (burning them for example – there are many factors here – the recycling process itself might consume a larger amount of energy – see a similar issue with the fact that placing solar panels in the Sahara desert might actually increase global warming), reusing is always a laudable goal.

The company is also relatively media-savvy: they have a (Facebook) like button on their website, they have a blog and their site is relatively nice looking (even more important: it doesn’t look like one of the stock templates from designers). The only thing you can’t do is to order online, but probably it’s better this way, since we are talking about relatively large amount of material (and automatically large amount of money) so a personal contact is a better option. Also, I didn’t see any indication that they ship outside of the USA, probably for the same reason.

Thumbs up for a small-medium business which has a good product!

Full disclosure: this is a paid review from ReviewMe. Under the terms of the understanding I was not obligated to skew my viewpoint in any way (ie. only post positive facts).

Medicalhealthinsurancetoday.com review

Attila-Mihaly Balazs — Fri, 08 Apr 2011 17:44:00 +0300

Today I’m reviewing a site which has the goal of comparing different private health insurance companies and giving you the cheapest one. Unfortunately I’m not familiar with the USA insurance rules (because I’m on a different continent :-)), so I can only comment on generic impressions related to this site:

The domain was registered in 2007, which is reassuring, however the registration information is hidden, which raises some questions, especially given the fact that you are supposed to trust this site with personal data (like address, phone number, date of birth, name)
The design is ok, although there are some technical glitches (like using the sitemap link to give the sitemap for the search engines, although this wouldn’t be necessarry - there are other ways to point the search engines to it)

However the biggest downfall of the site is the confusing interaction model and stale data: when you request a free quote it asks you a lot of personal information (then again I don’t know how much data the individual insurance companies risk model needs) in a separate popup. It presents the result in both the popup window (however both links it gave me gave me an 404 error) and also the main window, but clicking trough the main window requires to fill the form again.
In conclusion I have a low confidence level that such comparison sites would be a reliable information source and also the comparison on price alone isn’t enough when making such an important decision.

Full disclosure: this is a paid review from ReviewMe. Under the terms of the understanding I was not obligated to skew my viewpoint in any way (ie. only post positive facts).

Setting the maximum number of opened files under Ubuntu (for JProfiler)

Attila-Mihaly Balazs — Sun, 20 Mar 2011 14:29:00 +0200

As I found out “on my own skin”, setting fs.file-max in /etc/sysctl.conf is a BAD idea. It can render your system useless in one step. Please don’t do it! If you did it, use the recovery mode to roll back the change. Also, currently I would only recommend doubling the limit (ie going from 1024 to 2048 or from 2048 to 4096) not going to the maximum value.

JProfiler is a great tool, however under 32 bit Ubuntu you can run into the problem of having a too low limit for open filehandles. This is a problem for JProfiler because it uses temporary files to work around the address-space limitation created by 32 bit (yeah, I know, I should upgrade to 64 bit - but 32 bit works great for now…)

To raise the maximum filehandle limit, do the following:

sudo gedit /etc/security/limits.conf
# add the following two lines before the # End of file marker
# yes, the initial star is also part of line, and you should add it
*       hard    nofile  4096
*       soft    nofile  4096
sudo gedit /etc/sysctl.conf
# restart your system

You can check if the changes were successful by using the ulimit command:

ulimit -n
# it should print out 4096

DiskMap - an disk backed Map in Java

Attila-Mihaly Balazs — Tue, 08 Mar 2011 18:12:00 +0200

I have the following problem: a Java application was running out of memory. It was not feasible to mandate 64 bit JVM for this application and the \~1.4G limit wasn’t enough.

My solution was to implement a Map which - when an element is added - also saves the value to disk and only holds a weak reference to the value. When the memory pressure occurs, these objects, only linked by the weak references are evicted. Later, when they need to be read, they are read from the backing file.

Limitations:

Adding elements takes considerably longer (because they need to be serialized)
There is no way to reclaim space from the backing file (this is only intended for short-running mostly read-only tasks)
This is only useful if the values are considerably larger than the keys (because the keys are kept in-memory and only the values have the potential to be removed)
There is a memory overhead: when the objects are in-memory, you will take up an additional 20 to 40 bytes per entry. However, when the GC kicks in will only take up a 20 to 40 bytes per key.

Long story short: you can find the code (together with unit-tests) in my repo.

Why running sushi is the best fast-food?

Attila-Mihaly Balazs — Tue, 08 Mar 2011 16:44:00 +0200

I just realized that running sushi is the best fast-food ever. (Yes, I have strong opinions weakly held):

You get you food in small chunks, so you can stop at any time and still don’t feel like you’ve wasted food
You have a great variety of food and you can look at it before taking (rather than just looking at a picture in the menu and wondering how the real thing will look like)
It is most probably healthier than other kinds of fast-food
It is neither hot nor cold, so you can eat it right away (you don’t have to wait for it to warm up or to cool down)
You don’t have to order! The last thing you want to do when you are hungry is to stare at food and wait

PS. If you are Cluj (Romania) you can check out the Wasabi Running Sushi. As far as I know they are the only running sushi in Romania!

Microbenchmarking and you

Attila-Mihaly Balazs — Mon, 07 Mar 2011 15:13:00 +0200

Crossposted from the Transylvania JUG website.

Microbenchmarking is the practice of measuring the performance characteristics (like CPU, memory or I/O) of a small piece of code to determine which would be better suited for a particular scenario. If I could offer but one advice on this, it would be this: don’t. It is too easy to get it wrong and bad advice resulting from bad measurement is like cancer.

If you don’t want to take my first advice, here is my second advice: if you really want to do microbenchmarking watch this talk by Joshua Bloch: Performance Anxiety and use a framework like caliper, which I present below.

caliper is a Java framework written at Google for doing Java microbenchmarks as correctly as possible. To use, first you have to build it (there are no prebuild jars yet, nor is it present in the central Maven repository, sorry):

svn checkout http://caliper.googlecode.com/svn/trunk/ caliper
cd caliper
ant

Now you can start writing your benchmark. Benchmarks are written in a style similar to the JUnit3 tests:

you have to extend the com.google.caliper.SimpleBenchmark class
your methods must conform to the public void timeZZZZ(int reps) signature
you can override the setUp and tearDown methods to implement initialization / finalization

Below is a simple example (taken from the caliper homepage):

public class MyBenchmark extends SimpleBenchmark {
  public void timeMyOperation(int reps) {
    for (int i = 0; i < reps; i++) {
      MyClass.myOperation();
    }
  }
}

To run this you have multiple possibilities:

Use the caliper script included in the code distribution (this is a SH script, so it won’t work under Windows):
```
~/projects-personal/caliper/build/caliper-0.0/caliper --trials 10 org.transylvania.jug.espresso.shots.d20110306.MyBenchmark
```
you can also execute the script without parameters to get a list and description of command line parameters.
Run it from your favorite IDE. You need to add the following libraries: allocation.jar, caliper-0.0.jar. The main class is com.google.caliper.Runner and the parameters are the same you would pass to the caliper runner

Add a main method to your test class which would contain the following:

public static void main(String... args) throws Exception {
  Runner.main(MyBenchmark.class, args);
}

By default caliper outputs an easy to understand text output. You have also the option to publish the benchmark as a nice HTML page (see this page for example). The publication is done trough a Google AppEngine app and is publicly available to anyone (a caveat to remember). For more information see the caliper questions on StackOveflow. You might also be interested in the java performance tunning website if you need to perform such tasks.

Doing some estimations

Attila-Mihaly Balazs — Sun, 06 Mar 2011 10:39:00 +0200

This is again one of those topics which I like to rant about, so I give you the short version: when you see a number, question it! Most of the numbers thrown at us in different media can be disproven quite easily and it is our responsibility as people not to just repeat whatever we’ve heard, but rather stop and think a little about it (of course I’m not immune to this myself, since I’ve just fallen into this trap when reading the “Contemplating Financial Trading At Picosecond Resolution” on Slashdot, only to see the very insightful comment: light travels 3mm in a picosecond – yes I’ve done the math - so this article is pure BS).

Offtopic: why do sayings in different languages have so much in common? For example we have the “beating the dead horse” expression in English and in Hungarian we would say somebody is talking about is “horse made of branch” (vesszoparipa). Ain’t it interesting?

Getting back to my rant :-). I’ve seen an article recently about a local (Romanian) affiliate program: eMAG Profitshare 2010. I applaud them for their openness and it also gives us the possibility to do a quick calculation. They say that they’ve given out 463 000 RON (\~109 905 EUR / 153 499 USD) to 8690 sites.

Does it sound like a lot? Yes. Is it a lot for each individual site? Unlikely. Lets do a quick math: assuming that each site gets the same share (a very simplistic assumption) we have: 109 905 EUR / 8690 site = \~13 EUR per site / per year (these are yearly figures for 2010) so around 1 EUR (!) per site per month (!).

Ok, so be more real. You have a big fanbase, so you should be in the top sites as revenue. Lets consider a binomial distribution of the sites and do a little chart with Google Docs:

What you see here is the revenue per month for a site in a certain category (categories are from 1 to 10, 1 being the lowest traffic one and 10 the highest traffic one). The number is in EUR. The conclusion: this business model is a very poor revenue source for the individuals participating, but probably a very good marketing avenue for companies (I assume that the cost for companies is around the same as doing a ad campaign, but the returns must be much better – not to mention the google juice they must be getting from this referrals!).

PS: in the name of transparency, you can see the sheet I used for calculation here.

Audio quality

Attila-Mihaly Balazs — Sat, 05 Mar 2011 09:39:00 +0200

This is just one of those topics which comes up from time to time in my life (probably because I consume a lot of media). I was recently watching the Jim Zemlin interviewed by Jeremy Allison (Jim Zemlin is the Executive Director of the Linux Foundation) on the Google Open Source YouTube channel and was frustrated by the background noise and low audio volume, since the topic was really interesting to me. So I decided to look into the problem and see if the audio quality could have been easily improved. I covered the topic a couple of years so I won’t go into details, rather just give a 10 000 foot view of the process. Please read the original post for more details, since everything in it still applies.

Step 1: download the YouTube video. VLC natively supports YouTube playback, so exporting the sound to a FLAC file (you should always use lossless codecs during processing!) was just a matter of a couple of clicks and one or two minutes.

Step 2: load up in Audacity and remove the noise. The loading of the FLAC file is a little buggy (the progress bar keeps jumping between 0 and 100% and the time estimation is useless, but it loaded in under a minute). As you can see in the screenshot below, the volume is really low, but there are the occasional spikes, so plain normalization wouldn’t help you here. On the upside, there is no clipping which would result in a hard (impossible?) to repair artifacts.

After noise removal and keeping only one channel (no need for stereo here – we would add it back in the last step if we would to publish it since some devices can’t handle mono and the overhead with joint stereo is almost zero) the file was exported into WAV and fed into the Levelator. Here is the end result:

As you can see, we have much better volume resulting in a much improved experience for the consumer, all this with a couple of minutes of work while browsing Hacker News and with free (and mostly open-source) cross platform tools.

Content publishers of the world: please take a couple of minutes of your time after editing to do a proper post-production! Thank you.

Update: YouTube downloading is broken in the current VLC release but it will be fixed in the next version (1.1.12). Until then you can use the nighly builds.

Sorry for the malware warning!

Attila-Mihaly Balazs — Fri, 25 Feb 2011 12:02:00 +0200

If you have tried to visit my blog recently, you might have to a warning like this from your webbrowser:

Warning: Something's Not Right Here!

hype-free.blogspot.com contains content from randaclay.com
, a site known to distribute malware.
Your computer might catch a virus if you visit this site.
...

The source of the warning is the image / link in the comment form, which I have now removed (or more precisely replaced with a local copy). It seems that the randaclay.com has been hacked and thus it is classified as malicious by Google, which in turn leads to all sites linking to it being marked a potentially malicious. So, while I’m sorry for doing this, I will remove the links to their site until they manage to resolve the issue and will mirror their manifesto below:

Almost all blog platforms by default are set up so that a “dead end” piece of code is inserted wherever there is a link in a comment, so that search engines will not “count” the link as they are crawling the internet. This was originally designed to help stop comment spam, but it doesn’t work. What it does is remove some of the incentive for your readers contribute to your site by commenting on your posts.

What can you do about it? Turn off “nofollow”. Show your commenters that you appreciate them. Spread the link love.

![](http://1.bp.blogspot.com/-zrkpq0-gFoI/TWd7gbceiOI/AAAAAAAADVE/a8N_wMwWdrw/s1600/ifollowltgreen.gif)

mytrendyphone.co.uk review

Attila-Mihaly Balazs — Fri, 25 Feb 2011 11:48:00 +0200

My assignment - which I choose to accept :-) - was to review a mobile accessories site, namely mytrendyphone.co.uk. All the signs for this site check out:

It was registered a couple of years ago and didn’t move around much
It has several real seals (real meaning that they are linked to the originating site, where you can check out the referring site is really approved, didn’t just faked the seal)
Their contact address checks out (based on Google maps) as does their phone number (which is in the London area)
I found a couple of complaints about them on this site, however they seem to be very responsive to the negative complaints

After doing the basic safety research I went on to browse the site and I found many interesting items, for example this Jabra headset. What I like about about these models of headsets is the fact that they use standard minijack for headphones, which means that they can easily be replaced in case they stop working (and it is my experience that headphones are the first to go bad).

An other category which I found interesting was the tools section. While some of the tools are clearly overpriced, the more complex kits look interesting and I am tempted to buy something from there.

Full disclosure: this is a paid review from ReviewMe. Under the terms of the understanding I was not obligated to skew my viewpoint in any way (ie. only post positive facts).

Setting up IMAP with Yahoo! Mail

Attila-Mihaly Balazs — Tue, 22 Feb 2011 23:03:00 +0200

I’m a long time Yahoo Mail user. Just to illustrate how long I’ve been with them: when I joined the space available was a couple of MBs! I staid with them because I was mostly satisfied (never really caught the GMail bug), however recently I started looking for options to consolidate the different email accounts (work / personal / yahoo / gmail / etc). I explicitly wanted IMAP support because I really need to keep in sync between multiple machines.

The common wisdom seems to be on the ‘net that Yahoo! Mail doesn’t support IMAP (not even for paid accounts) or that various hacks are needed to support it (like sending custom / non-standard commands after login). This information however seems to be outdated, since I was able to find a least 3 IMAP servers (I’ve tested them all and they all work - with standard email clients with no hacks!):

imap.mail.yahoo.com (this is the one Thunderbird configures by default)
winmo.imap.mail.yahoo.com (from this article)
zimbra.imap.mail.yahoo.com

All of the servers support SSL/TLS encryption, so they are safe to access even from public hotspots. The outgoing server is smtp.mail.yahoo.com, which also supports SSL/TLS (and you should use it!)

The easiest to set up is Mozilla Thunderbird, however Evolution seems to work much better. One important feature in particular is that it works with large (10 000+ emails) folders, while Thunderbird chokes with an error (“UNAVAILABLE] UID FETCH too many messages in request”). To have Evolution work properly, you need to select “IMAP+” (also called IMAPX) as the protocol.

HTH somebody out there.

Manually enabling IP routing in Windows XP

Attila-Mihaly Balazs — Sun, 06 Feb 2011 11:37:00 +0200

While Internet Connection Sharing is a nifty tool, there are some cases where you would like to do the steps manually. One such case would be if the “primary” network is already using the 192.168.0.1/24 address space, since ICS is hardcoded (as far as I can tell) to use the same network. One concrete case I have encountered was:

ADSL Modem+Router (no wireless) –-> laptop broadcasting over writess –-> ... –-> other laptops

The solution is the following:

Set up an ad-hoc network on laptop wifi card (using a different subnet – 10.0.0.0/24 for example)
“Connect” to it to make it work:
Enable IP forwarding on the machine from the registry

It is simple as 1-2-3 :-p. Some caveats though:

This setup won’t give you DHCP. So make sure that you configure your other machines with a static IP address
It also won’t give you DNS, so configure something like the Google DNS (8.8.8.8 or 8.8.4.4) or OpenDNS (208.67.222.222 or 208.67.220.220) or even your ISPs DNS
The ad-hoc wifi connection has reliability issues. It happened multiple times that I had to restart it because it disconnected and wouldn’t connect any more, but it is a good temporary solution.

PS. You can download the drivers and user manual for the SmartAX MT882 ADSL Router here (the link might go dead unexpectedly, since it is served out of Dropbox). This is a standard modem provided by Romtelecom (the Romanian telecom provider) and I couldn’t find it elsewhere because Huawei is very secretive about its stuff (the files were copied from the CD provided with the modem). The driver makes the USB connection work as a network card (which is very elegant and simple).

Is hand-writing assembly still necessary these days?

Attila-Mihaly Balazs — Sun, 06 Feb 2011 09:14:00 +0200

Some time ago I came over the following article: Fast CRC32 in Assembly. It claimed that the assembly implementation was faster than the one implemented in C. Performance was always something I’m interested in, so I repeated and extended the experiment.

Here are the numbers I got. This is on a Core 2 Duo T5500 @ 1.66 Ghz processor. The numbers express Mbits/sec processed:

The assembly version from the blogpost (table taken from here): \~1700
Optimized C implementation (taken from the same source): \~1500. The compiler used was Microsoft Visual C++ Express 2010
Unoptimized C implementation (ie. Debug build): \~900
Java implementation using polynomials: \~100 (using JRE 1.6.0_23)
Java implementation using table: \~1900
Built-in Java implementation: \~1700
Javascript (for the fun of it) implementation (using the code from here with optimization – storing the table as numeric rather than string) on Firefox 4.0 Beta 10: \~80
Javascript on Chrome 10.0.648.18: \~40
(No IE9 test – they don’t offer it for Windows XP)

Final thoughts:

Hand coding assembly is not necessary in 99.999% (then again 80% of all statistics are made up :-p). Using better tools or better algorithms (see the “Java table based” vs. “Java polynomial”) can give just as good of performance improvement. Maintainability and portability (almost always) trump performance
Be pragmatic. Are you sure that your performance is CPU bound? If you are calculating a CRC32 of disk files, a gigabit per second is more than enough
Revisit your assumptions periodically (especially if you are dealing with legacy code). The performance characteristics of modern systems (CPUs) differ enormously from the old ones. I would wager that on an old CPU with little cache the polynomial version would have performed much better, but now that we have CPU caches measured in MB rather than KB the table one performs much better
Javascript engines are getting better and better.

Some other interesting remarks:

The source code can be found in my repo. Unfortunately I can’t include the C version since I managed to delete it by mistake :-(
The file used to benchmark the different implementations was a PDF copy of the Producing Open Source Software book
The HTML5 implementation is surprisingly inconsistent between Firefox and Chrome, so I needed to add the following line to keep them both happy: var blob = file.slice ? file.slice(start, len) : file;
The Javascript code doesn’t work unless it is loaded via the http(s) protocol. Loading it from a local file gives “Error no. 4”, so I used a small python webserver
Javascript timing has some issues, but my task took longer than 15ms, so I got stable measurements
The original post mentions a variation of the algorithm which can take 16 bits at one (rather than 8) which could result in a speed improvement (and maybe it can be extended to 32 bits)
Be aware of the “free” tools from Microsoft! This article would have been published sooner if it wasn’t for the fact MSVC++ 2010 Express require an online registration and when I had time I had no Internet access!
Update: If you want to run the experiment with GCC, you might find the following post useful: Intel syntax on GCC

Picture taken from the TheGiantVermin’s photostream with permission.

How to quickly start up a webserver with Python

Attila-Mihaly Balazs — Sun, 06 Feb 2011 08:14:00 +0200

Sometimes you need to quickly start up a webserver that serves up static files (I will describe such a case in the next post). Python to the rescue (works on both Linux and Windows if you have Python installed):

For Python 2.x (this is what most sites show you):

python -m SimpleHTTPServer 9914

For Python 3.x (thanks to this comment):

python -m http.server 9914

These will start up a webserver on port 9914, and you can access it via the address http://localhost:9914. Warning! The webserver will be available to anyone who can connect to your computer directly (unless there are other mechanisms to restrict it – like firewalls or NAT), so use your judgment!

Picture taken from bob’s photostream with permission.

Why you should use 0.0.0.0 in your hosts file - redux

Attila-Mihaly Balazs — Sat, 05 Feb 2011 20:50:00 +0200

Some time ago I (wow, time files!) I suggested that using 0.0.0.0 for host-file based blocklists would be faster than using 127.0.0.1. Above you can see an other reason for using 0.0.0.0: some applications take up port 80 on the localhost and accessing it can (potentially) create havoc.

In the example above TeamViewer (which is quite a nice remote control application BTW – with support for Linux!) has taken it over and thus it displays a mock page instead of the advertisement (which is very courteous).

PS. This was also mentioned in the original article, I just wanted to give an other example.

Augmenting Log4J stack traces with class versions

Attila-Mihaly Balazs — Thu, 03 Feb 2011 18:07:00 +0200

If you have multiple versions of your code in production, it is extremely useful for the log to include the version of the classes when producing a stacktrace, otherwise it is very hard to match the lines in the stacktrace with the lines of the source code (sidenote: there is an optimization in the Sun JVM where - if an exception is thrown “too much” - the JVM stops providing stacktrace - see this article about it and to learn how to disable this feature).

If you are using Log4J as your logging framework, with a little magic you can turn the following stacktrace:

17:47:40,208 ERROR [main] [TestLog4jExtendedStacktrace] An exception has occurred!
java.lang.IllegalArgumentException: Test exception
 at ...Callee.called(TestLog4jExtendedStacktrace.java:144)
 at ...Caller.call(TestLog4jExtendedStacktrace.java:136)
 ...

Into this (note the class versions at the bottom):

java.lang.IllegalArgumentException: Test exception
 at ...Callee.called(TestLog4jExtendedStacktrace.java:144)
 at ...Caller.call(TestLog4jExtendedStacktrace.java:136)
 ...

Callee: $Revision: 56 $
Caller: $Revision: 56 $

The main mechanism is as follows:

Each class contains a static String field called “VCS_VERSION”. This field is set (trough the magic keyword substitution - see the documentation for SVN, CVS) to the last version when the file was committed to your VCS.
When a stacktrace is sent to the logger, it will look at the classes and if they have the “VCS_VERSION” field, it will output it at the end (it doesn’t annotate the stacktrace itself, because some tools - like IDEs - depend on the stacktrace having a certain format for them to be able to process it - like adding one-click “go-to-line” shortcuts)

As usual, the source can be found in the repository.

Some more implementation details: the current way of modifying the stacktrace is quite hackish (adding a filter and trough reflection modifying the passed in LoggingEvent object). However it has the advantage of being usable without modifying your config files (ie. if you have many config files, but can modify it in only one place in the code, this is a good solution. More “proper” alternatives would be to implement a “wrapper” appender (like AsyncAppender) or a wrapper around Layout, however both of these require you to modify your configuration files.

Remote debugging with Java

Attila-Mihaly Balazs — Tue, 25 Jan 2011 15:35:00 +0200

Sometimes you have the situation that an issue is only occurring on certain machines or only at a certain time of day. There are a couple of possible methods to investigate such an issue (like: adding extra logging), however I would like to add an other one: remote debugging trough TCP/IP.

To do this, start your java program with the following jvm paramters:

-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=23334

The meaning of the parameters is as follows:

server=y – this application will act as a TCP/IP server (“acceptor”) and wait for incoming connections rather than trying to connect to you
suspend=n – the server will not suspend on startup (alternatively you can set it to “y” in which case it will pause and wait for the debugger to connect – useful if you need to debug issues occurring at startup)
address=23334 – the port on which the debugger will listen. Keep in mind that only one program can listen on a given port on a machine and if the given port is not available, the given program will not start

After the program has started open your Eclipse, go to Debug configrations, Remote Java application, create a new entry and set “Host” to the machine name or IP and “Port” to 23334 (or whatever other port you’ve set up). Connect to it and off you go. The configuration steps for IntelliJ can be found here (I didn’t check it, but they seem right). A couple of final thoughts:

If your sources are not in sync with the remote jars, you will see weird stuff (like breakpoints not triggering, triggering and the “wrong” line, etc), so you should make sure that you have the same sources as the jar does. If you still get into the situation where the sources are different from the classfiles, I found that setting breakpoints on “method entry” works as expected (ie. it breaks even if the method in the classfile is on a different line)
You can “detach” from a certain process and it keeps running (and later on you can re-attach to it)
This method is of low bandwidth / overhead, so it can be used to debug servers in remote locations
Never, ever do this in production! unless you are absolutely, 100% certain that you know what you are doing.

Navigating (Searching) Collections

Attila-Mihaly Balazs — Tue, 25 Jan 2011 15:04:00 +0200

Update: this article has been crossposted to the Transylvania JUG blog.

The Java collections framework includes the concept of NavigableSets / NavigableMaps. The principle behind these interfaces is that taking a SortedSet/SortedMap you can use a subset of it. Some examples:

Given the following set:

@Before
public void setUp() {
  set = new TreeSet();
  set.addAll(Arrays.asList(1, 2, 3, 4, 6, 7, 8));
}

The following is true:

// Returns the least element in this set greater than or equal to the given element
assertEquals(Integer.valueOf(6), set.ceiling(5)); 
// Returns the greatest element in this set less than or equal to the given element
assertEquals(Integer.valueOf(4), set.floor(5));
// Returns the least element in this set strictly greater than the given element
assertEquals(Integer.valueOf(7), set.higher(6));
// Returns the greatest element in this set strictly less than the given element
assertEquals(Integer.valueOf(3), set.lower(4));

// Returns a view of the portion of this set whose elements are strictly less than toElement.
assertTrue(set.headSet(4).containsAll(Arrays.asList(1, 2, 3)));
assertEquals(3, set.headSet(4).size());
// Returns a view of the portion of this set whose elements are greater than or equal to fromElement.
assertTrue(set.tailSet(4).containsAll(Arrays.asList(4, 6, 7, 8)));
assertEquals(4, set.tailSet(4).size());
// Returns a view of the portion of this set whose elements range from fromElement, inclusive, to toElement, exclusive.
assertTrue(set.subSet(4, 8).containsAll(Arrays.asList(4, 6, 7)));
assertEquals(3, set.subSet(4, 8).size());

Also, the subsets / submaps / “views” remain connected to the parent collection, so adding / removing to/from the parent collection updates them:

SortedSet headSet = set.headSet(4);
assertTrue(headSet.containsAll(Arrays.asList(1, 2, 3)));
assertEquals(3, headSet.size());

// subsets remain connected
set.removeAll(Arrays.asList(1, 2));
assertTrue(headSet.containsAll(Arrays.asList(3)));
assertEquals(1, headSet.size());

// subsets remain connected
set.addAll(Arrays.asList(-1, 1, 2, 3, 4, 5));
assertTrue(headSet.containsAll(Arrays.asList(-1, 1, 2, 3)));
assertEquals(4, headSet.size());

Finally, you manipulate the subsets and the result will be reflected in the original set (however if you try to add an out-of-range element, you will get an exception):

SortedSet headSet = set.headSet(4);
headSet.add(-1);
assertTrue(headSet.containsAll(Arrays.asList(-1, 1, 2, 3)));
assertEquals(4, headSet.size());
assertTrue(set.containsAll(Arrays.asList(-1, 1, 2, 3, 4, 6, 7, 8)));
assertEquals(8, set.size());

The implementation is very memory efficient, there is no copying of elements going on. One thing to consider is that by default these operations are not thread safe! Ie. if you generate two subsets of the same set and process them on two different threads, you must take care to properly synchronize the processing.

The complete source code can be found on Google Code under Public Domain or the BSD license.

How to test for the implementation of toString()

Attila-Mihaly Balazs — Tue, 25 Jan 2011 13:45:00 +0200

Update: This entry has been crossposted to the transylvania-jug blog.

Problem statement: you have some value objects for which you implemented toString() (for debugging purposes) and now you would like to test using a unit test that these implementations exist.

Possible solutions:

Use reflection to detect the existence of the method:
```
boolean hasToStringViaReflection(Class<? ?> clazz) {
  Method toString;
  try { toString = clazz.getDeclaredMethod("toString"); }
  catch (NoSuchMethodException ex) { return false; }
  if (!String.class.equals(toString.getReturnType())) { return false; }
  return true;
}
```
Advantage: no third party libraries needed. Also, no instance of the class is needed. Disadvantage: the actual code is not executed, so even trivial errors (like null dereferences) are not caught. Also, code coverage tools will report the lines as not covered.
Compare the string returned by the toString method to the string returned by Object and expect them to be different. This uses ObjectUtils from Apache Commons Lang:
```
boolean hasToStringViaInvocation(Object o) {
  return !ObjectUtils.identityToString(o).equals(o.toString());
}
```
Advantage: the actual code is executed, so trivial errors are detected. Also the code will be “covered”. Disadvantage: it requires an external library (however Commons Lang contains a lot of goodies, so it is sensible to add it most of the time). Also, it requires an instance of the class, so you need to be able to instantiate it.
Don’t use hand-coded methods at all, but rather some code-generation / AOP style programming like Project Lombok.

Again, these methods are to be used for toString methods which have debugging purpose only. In case the output of the method needs to conform to some stricter rule, more checks need to applied.

The complete source code can be found on Google Code under Public Domain or the BSD license.

Non-buffered processor in Perl

Attila-Mihaly Balazs — Tue, 25 Jan 2011 05:44:00 +0200

Lets say that you have the following problem: you want to write a script which processes the output of a program and writes out the modified somewere, with as little buffering as possible. One concrete example (for which I needed the script) is log rotation: you want to save the output of a program (which doesn’t support log rotation by itself) to a logfile which gets rotate at midnight (because it includes the date in the name). Also, an other constraint is that you would like to “time-out” the read attempt to do some maintenance work (for example you would like to rotate your logs – create the files with the different dates - even when no data is written to it).

One possibility would have been to use IO::Select, however it doesn’t support filehandles on Windows (not that Windows wouldn’t have the API to do so, it’s just that nobody was implemented it in Perl core). Fortunately we can have something very similar to it:

#!/usr/bin/perl
use strict;
use warnings;
use IO::Handle;

binmode STDIN;
binmode STDOUT;
STDIN->blocking(0);
STDOUT->autoflush(1);

my $BUFFLEN = 4096;
while (1) {
  my $buffer;
  my $read_count = sysread(STDIN, $buffer, $BUFFLEN);

  if (not defined($read_count)) {
    # nothing to read, pause
    sleep 0.1;
    next;
  }
  if (0 == $read_count) {
    # EOF condition
    exit 0;
  }

  syswrite(STDOUT, $buffer);
}

The magic is done here by STDIN->blocking(0); which sets the filehandle into a non-blocking mode, returning “undef” is there is nothing to read. Whenever this happens (ie. there is no data on the input) it pauses for a brief moment (1/10 of a second) and then retries.

Some other remarks about the code:

the input is read and the output is written as binary. This means that no processing is done which could screw up the flow (for example trying to convert data between character sets and screwing up Unicode characters)
care is taken to introduce minimal buffering. Output is produced as soon as the input arrives. For more intricacies of Linux buffering see this nice article at pixelbeat.
the code is very performant. I’ve measured throughputs up to 1.4 Gb/sec and can certainly handle anything the disk can (if we consider it in the context of log rotator)
the code has been tested and works on both Windows (Strawberry Perl 5.12.1) and Linux. It should work mostly anywhere since it uses Core Perl.

Comparative book review

Attila-Mihaly Balazs — Mon, 24 Jan 2011 17:42:00 +0200

Below is a a short comparative review of tow books about Java concurrency which I’ve read in the last couple of months. Disclaier: the Amazon links are affiliate ones.

Java Concurrency in Practice is an interesting book, which should be a must-read for anyone doing concurrent programming in Java (and in these days if you aren’t, you’re missing out on a whole lot of possible performance improvement). While some reader criticize it for the dense stile, it is hard to see how one could tackle such a complicated topic in simpler way (to paraphrase Albert Einstein: one needs to make things as simple as they need to be and no simpler). That said, the book definitely has the topics ordered from simple to more advanced, so even if you find the idea of reading the whole book daunting, you should look at the first couple of chapters at least. I would especially recommend chapter 3 (Sharing Objects) from part I (Fundementals) which should give a clear motive to everyone why they should be concerned by thread-safety and how they should reason about concurrent programs (I find that many concurrency errors occur because people have a naive and simplistic understanding of the way concurrency works on modern hardware).

Concurrent Programming in Java: Design Principles and Pattern (2nd Edition): The CPiJ book is much older, ancient even by computer age standards (published in 1999, compared to the JCiP book published in 2006). If also describes a much more manual, tedious way of doing things compared to the newer book. Also, it talks about the precursor of the java.util.concurrent package, since the package didn’t exists back then. All in all: if possible, get the JCiP book. If you already have the CPiJ book, it is a good introduction to the topic, however be ware that much of the advice is outdated and Java 6 (and even Java 5) contain better and simpler ways to perform the tasks described in the book.

noevir review

Attila-Mihaly Balazs — Mon, 24 Jan 2011 17:11:00 +0200

noevir is a “direct marketing” company focusing on cosmetics and “* care” (skin, body, etc) type of products. After looking at their site I’m mostly neutral about them. I wouldn’t recommend anyone to join such (“direct marketing”) organizations, but that’s not specific to noevir. It also says “Ginza Tokyo” in the header, which is a big shopping street in Tokyo, but I couldn’t find any other connection to Tokyo (nor did I see this brand advertised last summer when I was in Tokyo and I visited Ginza, but then again, I wasn’t looking for it). I also can’t find it on the BBB site (a worrisome sign), but the contact address conincides with the domain registration (a good sign) and it is a real address, findable on Google Maps. There are also a lot of negative articles on the web, but they are related to the “direct marketing” part of the business (ie. if you join as a consultant) not to the products. I couldn’t find anything negative about the products.

My final verdict is: use a temporary credit card if buying from them (always a good idea when dealing with smaller merchants). If you buy something, buy it for its obvious qualities (like its scent), not for the advertised but hard to quantify qualities (like “clensing”, “protection”, etc). If you need help with a skin issue, consult a medic.

Full disclosure: this is a paid review from ReviewMe. Under the terms of the understanding I was not obligated to skew my viewpoint in any way (ie. only post positive facts).

scentsy review

Attila-Mihaly Balazs — Mon, 24 Jan 2011 16:37:00 +0200

scentsy has an interesting concept for providing different scents in the room: rather than burning different materials (like candles or sticks) it uses a lightbulb to heat the wax. This provides a “smoke-free” way to enjoy your fragrances. An other advantage of the concept is that it keeps the warm glow of the candle. If scents are your thing and you don’t like the smoke part, give this a try. Take care however that some people are sensible to strong scents and they can have adverse reactions (like headache). The electric system can help here also: you can use an electric timer (easy to find is most places) to control the dosage. This is also helpful if you are concerned (as I am) about electric appliances overheating.

A couple of final thoughts:

it uses its own checkout system rather than something more known like Google Checkout. I would recommend using a one-time creditcard (like the ones offered by PayPal) for added safety (I just don’t feel comfortable giving my creditcard details to small merchants)
it has an A rating on BBB (which is good)
it also has some kind of a referral system. I would strongly advise people against participating in such systems, but if the products are good, buy them.

Full disclosure: this is a paid review from ReviewMe. Under the terms of the understanding I was not obligated to skew my viewpoint in any way (ie. only post positive facts).

Processing clipboard data in Perl

Attila-Mihaly Balazs — Mon, 03 Jan 2011 17:03:00 +0200

The problem: lets say you have a program which generates data to the clipboard (or it is easier to get the data into the clipboard than into a file) and you want to process the data (create a summary for example).

Perl to the rescue!

Get the Clipboard module (if you use Linux, it is as easy as sudo cpan -i Clipboard; sudo apt-get install xclip but the package is also available as an ActivePerl package for example).

Write a script like the following:

use strict;
use warnings;
use Clipboard;

my $clippy = Clipboard->paste();
my ($sum, $cnt) = (0, 0);
while ($clippy =~ /Processed in: (\d+)/g) {
        $sum += $1;
        $cnt += 1;
}

print $sum/$cnt, "\n";

Profit!!! :-)

Update: you can combine this with syntax highlight for example to obtain nicely formatted source code.

Update: copying stuff to the clipboard doesn’t seem to work under Linux (tested under Ubuntu 10.10) because it invokes xclip with the “primary” clipboard but it only seems to work with the “clipboard” clipboard. Unfortunately I didn’t find any good material about the distinction between these different clipboard types, but the “monkey patch” below fixes the problem for me (of course I also filed a bug with the package so this should be resolved in a future version).

use strict;
use warnings;
use Clipboard;

if ('Clipboard::Xclip' eq $Clipboard::driver) {
  no warnings 'redefine';
  *Clipboard::Xclip::all_selections = sub {  
    qw(clipboard primary buffer secondary)
  };
}

# ... your code here ...
Clipboard->copy('foofooo1');

Why Ubuntu 10.10 is better than Windows XP?

Attila-Mihaly Balazs — Mon, 03 Jan 2011 16:37:00 +0200

I want to preface this with the following: I don’t want to pull a fanboy move here. The only thing I assert is that a recent OS (ie. Ubuntu 10.10) can give a considerable performance improvement (without changing the hardware) compared to an almost 10 year old OS (Windows XP).

Without further ado, compiling a large(ish) Java project on Windows XP:

real    3m16.776s
user    0m2.333s
sys     0m0.796s

And Ubuntu 10.10:

real    1m32.169s
user    2m10.488s
sys     0m12.677s

More than twice as fast! Neat!

Update: a friend just got a newer machine with better processor (Core i5 vs Core Duo) with Windows 7. The new machine with Windows 7 compiles the project in \~1m50s, so still Ubuntu seems to be the better choice.

Java has some surprising amount of dinamism in it

Attila-Mihaly Balazs — Sun, 02 Jan 2011 18:15:00 +0200

Not long ago I saw some java code from Simone Tripodi. It generates synchronization wrappers around arbitrary objects at runtime in a typesafe manner with a couple of easy to understand lines of code. The heavy lifting is done by the dynamic proxy mechanism available from Java 1.5 if I recall correctly.

The downside is that there seems to be a 30% to 40% performance impact based on some quick benchmark I’ve done. However one can not understate the value of not having to write or maintain code!

There are surprisingly many things one can do in Java in a typesafe manner (including things usually associated with dynamic languages) which helps catching more errors early (at compile time) and take full advantage of the different helper features available in IDEs (such as auto-complete).

Java Date objects can mutate, even when read

Attila-Mihaly Balazs — Sun, 02 Jan 2011 10:43:00 +0200

Ran into this problem a couple of months ago, when we saw some strange dates in production. So I dug into the Java library sources (thank you Sun for providing those!) and found that Date objects aren’t always “normalized”. Rather, sometimes a “denormalized” value is stored which is later (lazily) normalized. The normalized value isn’t properly synchronized with regards to the Java memory model however, which means that sometimes you can get weir (and incorrect!) results.

To illustrate the problem, I’ve created a small program. It does the following:

It creates a Date object and sets it to certain values
Schedules multiple Runnable’s which examine the value of the object on a threadpool

Everything looks fine and dandy, right? The object isn’t changed (apparently) after being handed of to the threadpool, yet sometimes wrong answers still appear (it takes around \~30 min on my laptop for such an event). So what are the lessons here?

Get your API right! If the user doesn’t seem to be doing writing, don’t do writing!
You can still do lazy initialization (if you really want to), but be sure to make it thread-correct (volatile, synchronized, etc) or at least document it (even though nobody reads the documentation)
Source code FTW! I couldn’t have debugged this without source code. Ok, maybe I could (decompiling class files is not that hard), but probably I wouldn’t have bothered.
Finally, the solution (hack) in this particular situation is to call getTime() after setting the values, which preemptively normalizes the internal representation. Of course the proper solution would be to pass around truly immutable objects (like timestamps or value objects from Joda Time).

TigerChef review

Attila-Mihaly Balazs — Thu, 09 Dec 2010 22:23:00 +0200

Lately I’ve been experimenting with some simple cooking, and of course for cooking you need cooking equipment. TigerChef is one of the many sources of cooking equipment you can get, although they are oriented more at restaurants than individual needs.

Anyway, I’m quite happy with the initial experiments and with the help of my dear wife who endures them and gives feedback on it I hope to start making more and more complex dishes. It seems incredible how simple it is to put together a dish following a recipe (but it is also hard work). And you get a (hopefully) delicious reward at the end.

The two things I cooked so fare are potatoes cooked in their “skin” and “turosgomboc” (Google translate is no help to me finding the translations of these terms). The potatoes were the simpler of the two (you just had to cook them in the oven for 40 minutes and eat it with butter or cheese) while the “turosgomboc” was quite an involvement. Fortunately I found this recipe and it worked out great.

Full disclosure: this is a paid review from ReviewMe. Under the terms of the understanding I was not obligated to skew my viewpoint in any way (ie. only post positive facts).

Problems (and a semi-solution) for tcpdump with DAG cards

Attila-Mihaly Balazs — Fri, 13 Aug 2010 04:30:00 +0300

Documenting here for posterity, since I didn’t find much information about it on the ‘net:

Disclaimer: I’m not a network head, just an amateur who dabbles with it when he needs to fix a problem.

Given one Ninjabox (the nickname for packet capture boxes from Endace) with a DAG card (some kind of custom packet capture network card from the same company), you could get the following error when trying to use tcpdump on the dag interface:

tcpdump: dag_attach_stream: Permission denied

The problem seems to be unrelated to your privilege level (you will get this even if you are running as root), but rather to the fact that some other program is/was using the particular interface. You can quickly check this by doing a lsof | grep dag0. In my case it was softflowd. But even after killing the softflowd process, I was getting the same error message. I had to reset the card using the following commands:

/etc/init.d/dag_drivers_load stop
/etc/init.d/dag_drivers_load start

After this tcpdump worked like a charm. Hope that this information will save people from searching around as I had to do.

Off topic minirant: why use custom hardware / software? In my experience they almost never deliver the performance they promise and are hard to troubleshoot because of lack of information.

shuttledirect review

Attila-Mihaly Balazs — Fri, 13 Aug 2010 03:44:00 +0300

The purpose of this website is to offer prebookable Airport taxi transfers. This means that you can rent a transportation method to/from the airport before you even embark on your journey.

Is it worth it? I have to admit that I’m far from being a big traveler, but until now every airport I’ve been to offers many ways to travel to the nearest city which are clearly marked. Also, this site is an aggregator of local offerings (it doesn’t provide the services itself), which means that the quality may vary between different parts.

The trustlevel I have in the site (since security is a feeling, not some objective thing for most people) is medium for several reasons:

They exist since a long time (since 2002), which is good, but don’t have the proper SSL setup (which is bad, especially for a site which wants to take my money)
They don’t run on a dedicated server, although probably the other sites hosted on the same server belong to the same company (so not as bad as using shared hosting)
The different security logos which they present are not linked to the respective organizations (you can’t click on them to access the information the third-party has about the site)

Considering all the above, plus the fact that I never felt the need for such a service, I am somewhat reticent to recommend it, but it might work for other people.

Full disclosure: this is a paid review from ReviewMe. Under the terms of the understanding I was not obligated to skew my viewpoint in any way (ie. only post positive facts).

Copyright is not theft!

Attila-Mihaly Balazs — Wed, 09 Jun 2010 15:36:00 +0300

Recently there have been quite a few copyright-related posts which came up in my feedreader. This is of course a complicated and layered problem which can’t be solved in the couple of paragraphs of this blogpost, but at least I can post a bunch of great materials which should contribute to the edification of all of us.

From comixtalk.com: Copyright Is Not theft

http://www.youtube-nocookie.com/v/IeTybKL1pM4&hl=en\_US&fs=1&color1=0x234900&color2=0x4e9e00 Also from comixtalk: [Nina Paley Discusses State of Sita Sings The Blues](http://comixtalk.com/xerexes/nina_paley_discusses_state_sita_sings_blues). This is an animated movie (which you can [watch for free](http://www.archive.org/details/Sita_Sings_the_Blues)) that had legal problems because of the backing soundtrack, even though the music in question was created in 1920, so it should be in the public domain.

You might also be interested in the documentary [rip! a remix manifesto](http://ripremix.com/) (embedded below for your convenience). It is a documentary (Michael Moore style) talking about the issue. And while it isn’t perfect, it manages to raise a lot of interesting issues (BTW, personally I find the songs on the [Grey Album](http://en.wikipedia.org/wiki/The_Grey_Album) much better than the originals on the Black album - just a random example how derivative works can improve the original):

Finally here is a presentation from TED comparing industries with different level of copyright protection (via [Slashdot](http://news.slashdot.org/story/10/05/25/2222207/The-Fashion-Industry-As-a-Model-For-IP-Reform)):

http://www.youtube-nocookie.com/v/zL2FOrx41N0&hl=en\_US&fs=1&color1=0x234900&color2=0x4e9e00 This should be enough information to keep you outraged for weeks :-) PS. Just a quick rundown of my current opinion: all [works are](http://hype-free.blogspot.com/2009/09/myths-of-innovation.html) [derivative](http://hype-free.blogspot.com/2009/08/myth-of-cognitive-quantum-jumps.html). But even if we skip over this, long copyright stifles innovation. And even if we don’t consider (or don’t accept) this premise, labelling *all* copying as “theft” is (depending) wrong, (possibly purposefully) misleading and unethical. For example I posses the copyright for all the materials published on this blog (since it is my original work), but I explicitly grant anyone to reuse the content under the conditions of the [CC-BY-SA 3.0](http://creativecommons.org/licenses/by-sa/3.0/) license. PS \#2: An other interesting documentary to watch is [Patently absurd](http://patentabsurdity.com/). I didn’t include it above because it deals with patent law, not copyright, two domains which are frequently bundled together under the term “Intellectual Property” (together with trademark law), but the fact is that these three domains are completely separate and the laws governing them are distinct, ergo I didn’t want to add to the confusion. PS \#3: Technology != Breaking the law. Just because I use bittorrent, it doesn’t mean that I’m breaking the copyright law! I might be very well downloading a Linux ISO (as I frequently do), one of the many *free* (as in freedom) material from [Clearbits](http://www.clearbits.net/) (previosuly legaltorrents) or a World of Warcraft patch for that matter.

Dear people: try to think harder, even if it makes your head hurt!

Attila-Mihaly Balazs — Mon, 07 Jun 2010 05:36:00 +0300

This again is the case of a couple of links on the same topic piling up in my reader (this tends to happen if you take a pause in blogging :-)):

The commonality between all these articles is that they make statements based on faulty questions (PHD Comics says it best). A website poll is not the same as a scientific study (to name just one the problems - it has a selection bias towards the reader of the particular site - which wouldn’t be a problem if the results wouldn’t be presented as applicable for the general population). And even if they were scientific studies, the purpose of a scientific study isn’t to find the absolute truth! It is to present a hypothesis which doesn’t contradict any of the current observation. But it doesn’t exclude the possibility that in the future there will be an observation which contradicts the hypothesis, and as such, it must be changed.

Who is hype-free?

Attila-Mihaly Balazs — Sun, 06 Jun 2010 11:14:00 +0300

I’ve done a writeup about the name of blog when I started it. However recently two links came up in my Google Alerts:

The first one is from Urban Dictionary and it defines hype free as “Slang word for drug free”.

The second one is from Yahoo! Answers and states pretty much the same thing (in a conversational form).

I didn’t know this meaning of the expression until these items came on my radar recently, but they are very funny (and certainly true in more than one sense).

Update: the Yahoo Answers page states that it was deleted “according to our Community Guidelines”. Bummer.

On the hopelessness of pulling content from the interwebs

Attila-Mihaly Balazs — Mon, 10 May 2010 19:12:00 +0300

In the last couple of weeks I had at least two cases where I saw a (provocative) post come up in my feedreader, click trough to read the entire piece (BTW, partial feeds just suck!), just to find that the owner removed the post. The first was from the DynDNS blog named “Open Dialogue” (apparently openness and censorship can co-exists in some people’s minds, without having their brains blown-up by the cognitive dissonance) and it said the following:

We hope we’re wrong, but it looks like DNS Made Easy (aka Tiggee LLC) is secretly behind DNSComparison.com

Let’s first start off by providing some definitions of key attributes that Dyn Inc. lives by across our organization and takes pride in while representing the DNS industry. These characteristics define us and make us the company we are today. Call us naïve, but we also “still” hold out hope that the rest of the DNS space (and the business world, in general) believes the same and truly means well when their actions might seem otherwise.

The second one comes from the MaraDNS blog (is there a pattern here? are there many frustrated people in the DNS space? :-p):

Xonotic: Type 2 Freetards can’t make content

If you want to piss a type 2 freetard off, take an open-source project, make it proprietary (after getting everything with copyright to the code to agree to the non-GPL terms), and sell the proprietary product.
This happened with Tux Racer. Boy were the freetards pissed off, whining about how the commercial game wasn’t very good, blah blah blah. But, bottom line: The developers worked hard making the program. They wanted to get paid for their work. The type 2 freetards felt something was stolen from them because the next version of their program was not open-source.
Another successful open-source game is now doing the same thing: Nexuiz, an excellent fun little first person shooter with everything (both the engine and the content) under a GPL-compatible license.
Well, the developers realized one day that they wanted to get paid for their work, so they decided to have a remake of Nexuiz for consoles that will be closed-sourced using different content.
The freetards went ballistic. It became a front page story at Freetard central. In short order, a fork was declared. Freetards everywhere talked about how evil Nexuiz was; their declarations were mainly based on ignorance; inaccurate posts accusing the Nexuiz development team of violating the GPL were posted everywhere.
The next Nexuiz will, for the record, be 100% legal: All of the Nexuiz code has been licensed for non-free use. The content will be, for the most part, entirely new. There is no GPL violation here.
Once the dust cleared, development on the fork (called Xonotic) stalled. One developer recently admitted that, two months after declaring this fork, that

It is my opinion that such actions inherently undermine the trust in a person / brand. It is also ineffective (proven by the fact that at least one person – ie. myself – was able to read the content). My ideal publishing platform would be:

Versioned, so that everyone could look up what the text looked like at a given moment in time
Verified by a third-party agency (such as a timestamp signing service) which guarantees that it had a certain content at given point in time (you don’t have to transmit the full text to such a service BTW – them signing a cryptographic hash is good enough)
Digitally signed by the author

We all make mistakes. Lets act as grownups about it. Don’t try to wish some things away. I understand that in some circumstances there are legal obligations to take some things down, but at least post the takedown reason in these cases (ie. “this post was taken down because of allegedly defamatory content. sorry”).

Picture taken from sara\~’s photostream with permission.

Putting the eval into Java

Attila-Mihaly Balazs — Thu, 22 Apr 2010 17:42:00 +0300

“eval” (short for evaluate) is usually the name given to the method in dynamic languages which makes it possible for the programmer to access the compiler / runtime. Here are a few links to the documentation for the function in different languages:

They are usually used to quickly evaluate a DSL (Domain Specific Language) expression. What I mean by this is the following: lets say that the user supplies an expression which can be easily (ie. with a few string replacements or regular expressions at most) converted into a valid expression in the current language. Then you don’t have to write your own lexer / parser / runtime to support this function.

To make this example even more concrete, lets say that you are implementing a simple graphing calculator where the user can supply the right part of the f(x)=… expression and you draw the function for a given interval of x. If the user supplies something like 1 + 2*x + 3*x*x, this is pretty much a valid expression in all programming languages (there are minor syntactic differences to be precise - like Perl/PHP requiring you to prefix variable names by the “\$” sign), so you could simply use “eval” on it.

Warning! Running eval on unverified, user supplied code is a really, really bad idea! (yes, I know that red and bold underline is a little over the top, but this is just that bad! Never, never, ever do this! It is equivalent to letting everybody connected to the Internet (assuming that we are talking about an webapp) running arbitrary code on your server. Implement very strict filtering (based on whitelisting if at all possible) for such features!

Surely, you would say, such a dynamic feature isn’t easily accessible for a statically typed compiled language as Java… And you would be wrong! As of Java 6 each JVM install (including the JREs) includes the Java compiler, and it also includes a public API to access it. Using this feature you can implement the Java equivalent of “eval”: giving a string to the compiler and getting a class instance back, on which you can call methods. You can find the source in my SVN repo. It is (almost entirely) based on the following article from 2007 (just to give you an idea how long this option has been around): Create dynamic applications with javax.tools. An other (pleasant) surprise was the fact that this process doesn’t require any security privileges and works perfectly in restricted environments such as browser.

An additional advantage of using the JVM rather than your own runtime is speed: many man-hours have gone into optimizing both the source –> bytecode and the bytecode –> machine code transformations. Which brings me to an other possible use for this kind of solution: generating particularized instances of generic classes to give more hints to the JVM about possible implementations.

For example, the StrinkTokenizer class does the following when looking for separator characters:

char c = str.charAt(position);
if ((c > maxDelimCodePoint) || (delimiters.indexOf(c) < 0))
    break;

Now imagine how much more efficient (in the sense of: easier for the JVM to translate into an efficient machinecode) this code would be if we knew that we have exactly one possible delimiter (as it is the case most of the time). Replacing delimiters.indexOf(c) with delimiter == c can give you an order of magnitude speedup for this particular code.

The takeaway should be:

This is a very powerful technique, but it should be used with care! Only use this method if you’ve proven (by using a profiler for example) that the given class is the dominant factor in the performance picture.
Be particularly aware of potential security risks which could appear!
Also, be aware that you give up many things when going this route:
- Automated refactoring
- Reports generated by bytecode analysis tools (like coverage or bug detection)
- Debugger support
In conclusion: use it with great care, but if used properly, it can result in considerable performance improvements!

Picture taken from Hexadecimal Time’s photostream with permission.

Update to the Blogger Tag Cloud

Attila-Mihaly Balazs — Tue, 20 Apr 2010 17:27:00 +0300

A small PSE (Public Service Announcement): if you were using the Blogger Tag Cloud I’ve put together based on the WP-Cumulus plugin, you might have noticed that it stopped working some time ago (I’m not entirely sure when, since I didn’t notice it, until a reader commented and brought it to my attention – thanks again Soufiane).

The problem was that the server hosting the SWF and JS file didn’t serve them anymore, instead giving a 403 – access refused error. To mitigate this problem I’ve uploaded the SWF file to Google Code and used the JS file from the Google Ajax Library and bought the plugin back to life.

So, if you are using the plugin and you are subscribed to my feed, go to the original (now updated) post and use the new code.

Thank you and sorry for any inconvenience caused!

“Funny things I found while browsing the web” post

Attila-Mihaly Balazs — Fri, 09 Apr 2010 17:26:00 +0300

The Geek/Nerd/Dork/Dweeb Venn Diagram (via Joel Esler’s blog):

BTW, here is a quick way to convert JPEGs which should be PNGs or GIFs (because they aren’t photos!): simply use a photo editing software (like the GIMP or IrfanView / Paint.NET) and reduce the color depth without dithering. This should pretty much get you there. You might want to play around with the number of colors to retain.

The second one is Pixels by Patrick Jean (via Wondermark):

[dailymotion id=video/xcv6dv] [MC Frontalot](http://frontalot.com/) released his newest album Zero Day (via the [Veracode blog](http://www.veracode.com/blog/2010/04/mc-frontalot-releases-zero-day/)):

http://www.youtube-nocookie.com/v/5cBM4DdoC2A&hl=en\_US&fs=1&rel=0&color1=0x234900&color2=0x4e9e00

Updated YARPG

Attila-Mihaly Balazs — Fri, 09 Apr 2010 15:10:00 +0300

This has been sitting in my queue for some time: almost four years ago (it’s incredible how time flies!) – amongst the first posts I’ve published on the blog – I’ve written a random password generator in Javascript which I’ve named YARPG (for “Yet Another Random Password Generator”). The advantages to using it are the same as they were back then:

Customizable (password length, types of characters included, etc)
Secure (it doesn’t communicate over the network, hence no need for SSL)
Fully reviewable (as opposed to server-based solutions, where you have to trust the server)

The only flaw it had (as pointed out by a commenter) was the fact that passwords didn’t always include all the characters you’ve selected (ie. the checkboxes represented “possible” not “mandatory” characters, which was a little counter-intuitive).

I’ve thought about how to create passwords which included at least one character from each set. My first ideas were around generating a password, then checking that it contained at least one character from each set and if not, replacing some of the characters with ones from the missing set. However this train of thought quickly ran into problems when I had to decide which character to replace. Choosing something fixed (like the first one, last one, etc) is too predictable. If I choose a random one, I run the risk of overwriting previous change. So finally I realized that there is a simple solution: just re-generate the password until it satisfies all of the constraints. Although this might seem like a brute-force solution, in practice its speed is indistinguishable from a constant-time solution.

Below you have the new and improved YARPG:

Password length:
Include letters
Include mixed case
Include numbers
Include punctuation
Include all printable characters
Quantity

I’ve also updated the [original posting](http://hype-free.blogspot.com/2006/10/javascript-password-generator.html). You can get the source code for it by looking at the source of this webpage, or from my SVN repository: [js\_password\_generator.html](http://code.google.com/p/hype-free/source/browse/trunk/js_password_generator.html). Hopefully you find it useful! *Picture taken from [cjc4454’s photostream](http://www.flickr.com/photos/cjc4454/) with permission.*

Sending an X-Face email with Perl+GMail

Attila-Mihaly Balazs — Fri, 02 Apr 2010 16:17:00 +0300

In the latest Software Freedom Law Show Bradley mentioned the X-Face email header and challenged listeners to send them an email containing the X-Face header. So here is the small Perl script I’ve whipped together to send them an email trough GMail:

use strict;
use warnings;
use Net::SMTP::TLS;

my ($from, $password) = ('...@gmail.com', 'MySuperSecretPassword');
my $mailer = new Net::SMTP::TLS(
  'smtp.gmail.com',
  Hello => 'smtp.gmail.com',
  Port => 587,
  User => $from,
  Password => $password);

$mailer->mail($from);
$mailer->to('foo@example.com');

my $data = <<'EOF';
X-Face: "8.]Z_3ptu\NK'CA~DM>M,G.T(h=1.y9"0gXW3V91E:dw2?|&G2R(?/no'F2g4%8Fv.
 J1p5K-^1epKXxIG)mj4}nGWTi<=iz8n)bUVhLu}MXRFl9"J%'=-;IfMXcuPK>-%^;$uW87O/B
Subject: Hello X-Faced World!

email body.
EOF

$mailer->data();
$mailer->datasend($data);
$mailer->dataend();
$mailer->quit();

The code is largely based on this snippet: Sending Mail Through Gmail with Perl. The X-Face header was generated using the Online X-Face Converter (yes, I know that there is a Image::XFace module, but it was very cryptic – it didn’t mention supported input / output formats). One word of warning: if you are using ActivePerl under Windows, Net::SMTP::TLS isn’t available in the default module list (AFAIK, because of encryption restrictions), so you might need to experiment with alternative package sources or using Linux :-). I’ve also tested the script with an email account I control (using Thunderbird with the Mnenhy plugin – which can read but not create X-Face emails) and it worked nicely.

There you have it: how to use an old (from the 1980s according to Wikipedia) method for embedding pictures which is not supported by most of the email clients :-)

Performance optimization techniques for Java code

Attila-Mihaly Balazs — Fri, 26 Mar 2010 13:56:00 +0200

Yesterday I gave a presentation at the Transylvania JUG about using profilers and different techniques which you can use to work around the discovered performance problems. Below you can find the embedded presentation.

**http://static.slidesharecdn.com/swf/ssplayer2.swf?doc=java-perf-201003-100326052203-phpapp02&rel=0&stripped\_title=performance-optimization-techniques-for-java-code**

If you are interested in the code samples (as you most probably are, since a big part of the presentation were demos), you can find them in [my Google Code SVN repository](http://code.google.com/p/hype-free/source/browse/#svn/trunk/java-perfopt-201003). Feel free to contact me if you have questions about the slides or about something else in general.

Solving mathematical puzzles with brute-force and Perl

Attila-Mihaly Balazs — Fri, 26 Mar 2010 11:47:00 +0200

After talking a lot about optimizations and selecting the right algorithm, here is a little brute-force code. This particular one gives the answer to the following puzzle from Richard Wiseman’s Blog (one well worth following BTW):

Can you make the number 24 with the number 5, 5, 5, and 1 (again, you cannot join the numbers together, have to use each number once and only once, and are only allowed to add, subtract, multiply or divide them)?

And here is my brute-force solution:

permute(0, [5, 5, 5, 1], []);

sub permute {
  my ($partial, $numbers, $solution) = @_;

  if (0 == scalar(@$numbers)) {
    print @$solution, "\n" if (24 == $partial);
  }
  else {
    for my $num (@$numbers) {
      my $mynums = [];
      my $skipped = 0;
      for my $mynum (@$numbers) {
        if ($num == $mynum && !$skipped) {
          $skipped = 1;
        }
        else {
          push @$mynums, $mynum;
        }
      }

      for my $op (qw(- + * /)) {
        my $mypart = eval "$partial $op $num";
        my $mysol  = [@$solution, $op, $num];
        permute($mypart, $mynums, $mysol);
      }
    }
  }  
}

The output is not very elegant and contains a decent amount of garbage (because it considers that we have a hidden zero at the start – ie. 0*5…) and also a lot of repetition (because it doesn’t take into account that 5 5 5 1 is the same as 5 5 5 1 with the first two numbers interchanged), but in the end it gives the correct answer:

... fake answers because it starts with 0 ...
*5+5*5-1
*5+5*5-1
/5+5*5-1
/5+5*5-1
*5+5*5-1
*5+5*5-1
/5+5*5-1
/5+5*5-1
*5+5*5-1
*5+5*5-1
/5+5*5-1
/5+5*5-1
... duplicate answers because of the order ...
-1/5+5*5
-1/5+5*5
-1/5+5*5
-1/5+5*5
-1/5+5*5
-1/5+5*5

Also the correct interpretation of the output is to consider that each operation has a pair of parentheses around them and not reading it according to the usual mathematical rules. Having this in mind the solution becomes:

((-1/5)+5)*5 = 4.8 * 5 = 24

Brute-force FTW :-)

RHUB review

Attila-Mihaly Balazs — Thu, 25 Mar 2010 17:24:00 +0200

RHUB is a company which offers “meeting recording, Web Conference, Remote Access, Webinar software Delivered in Appliances”. It is an interesting model, especially given that there are a lot “software only” solutions in this space (GoTo meeting for example, just to name the biggest of them). They tout the simplicity of the product, especially that you don’t need any software installed to attend a meeting (~~I assume that it works using some kind of browser plugin – Flash or Java – and technically you do need that to be installed~~) Update: the folks from RHUB commented below and they pointed me towards their demo page. They seem to have a nice fallback mechanism, where by the main method involves Flash, but if it isn’t available, it falls back to Javascript + static images (although I assume that you loose most of the features, including voice, in that version). Very nice.

Simplicity would be a welcome addition to this area, since too many times have I seen “virtual” conferences spending tens of minutes on setting up the meeting (and it is quite frustrating).

For pricing they use the same model as the software-only solutions, namely a per-user license. If we compare the prices, you will conclude that the cheapest unit is equivalent to one and half years of GoToMeeting subscriptions, which is acceptable. I see this being a good solution for larger virtual training events (universities for example), however the value for smaller shops is not that clear. ~~I worry that using a physical device means that your entire staff looses connectivity when/if the device goes down (because of a power / ISP outage). If your workforce is distributed around the globe (or even in one country), “cloud” service could route around the problem for the people who still have Internet access.~~ Update: (again, from the comment below): the company seems to have a fallback solution for such outages, where by they offer services using their (hosted) equipment during outages, free of charge.

Full disclosure: this is a paid review from ReviewMe. Under the terms of the understanding I was not obligated to skew my viewpoint in any way (ie. only post positive facts).

String.intern() – there are better ways

Attila-Mihaly Balazs — Mon, 22 Mar 2010 13:04:00 +0200

I don’t want to write a “considered harmful” article (because they are harmful), but after experimenting with different solutions I do have a strong opinion that there almost no reason to use String.intern() in Java. But let us proceed step-by-step.

First of all, what does String.intern() do? Go read the Javadoc for it and also take a look at String interning of Wikipedia. The essence of it is that if you have two strings s1 and s2 such that s1.equals(s2), there will be only one copy of the string stored if they are interned. From this definition follow the two usecases for string interning:

You read a lot of repetitive strings from an external source (a flat file or DB for example) and you need to keep them all in memory. In this case interning the strings has the potential to save you a lot of memory.
You’ve determined (by profiling your application!) that String.equals is a hotspot for your application and you would like to replace those calls with the == operator.

If you have different reasons for looking at String.intern(), you should think twice about them before going down the route. If you’ve thought about long and hard, and you still think that String.intern is the best solution for you, but not for any reason mentioned above, please leave me a comment! (Also, read the rest of this post, since it might give you a better alternative).

So, having the above usecases in mind, what is the problem with calling String.intern?

It is quite CPU intensive. Calling new String("foo").intern() can be an order of magnitude (10x to 15x based on some of my measurements) slower than new String("foo").
You have to remember to do it everywhere. This isn’t so fatal if you’re just aiming for reduced memory consumption, but if you forget to call “intern” somewhere and later use the “==” operator for comparing elements, you can create some hard to track down bugs.
It can result in mysterious “OutOfMemory” exceptions. In the SUN JVM (which is the most widely used one) “internalized” String’s are stored in a special memory location called “PermGen”. The size of this isn’t influenced by the usual “-Xmx1024M” command line option, you have to remember (and to know about it in the first place!) to use the “-XX:MaxPermSize=512m” command line.

These are some very serious problems. What are the alternatives? The easiest one is not to use String.intern. Ok, lets say that you’ve performed measurements with relevant, production data and came to the conclusions that your problems need to resolved using this method. My recommendation would be the following:

Use a WeakHashMap to create a pool of Strings as describe in this blog post. This has the advantage that your cache won’t end up keeping the objects in memory after all the references to it have disappeared. Don’t forget to synchronize access to it if you’re planning on using it from multiple threads!
Always use String.equals, never “==”. If you take a peek at java.lang.String.equals, you will see that the first check that it does is “==”. By never using “==” explicitly you still will have most of the speed benefits, while eliminating the risk that you accidentally get a “rogue” String from somewhere and your code fails, even though the two strings are equal.

The advantages of the above solution are:

It is 30% to 50% faster than String.intern (although it is still slower than not calling String.intern. You should also watch out that it doesn’t become a chocking point in your application because of the synchronization if you are calling it from multiple threads).
It is safe (as mentioned above, forgetting to “make unique” some of the String’s doesn’t make your logic fail)
It doesn’t require special configuration on the JVM (like adjusting the PermGen size)

I will post some example code later this week when I’ll post the slides for a presentation I’ll be giving to the local JUG, so be sure to keep an eye on my blog and my code repository.

Some resources on the topic:

Busting java.lang.String.intern() Myths
interned Strings : Java Glossary (small trivia: substrings keep a reference to the original String, so if you’re keeping small parts of long String’s, you will be consuming much more memory than you might have anticipated)
Is java.lang.String.intern() really evil?

Picture taken from Mark Drago’s photostream with permission.

alreadyhosting.com review

Attila-Mihaly Balazs — Mon, 22 Mar 2010 11:11:00 +0200

There are many “web hosting review” sites out there and alreadyhosting.com is one of them doing reviews for Dedicated Server Hosting. These sites usually get their revenue by using affiliate links in their lists, and this particular one is no different. In general such lists are a good starting point, but you should always be cautious when choosing a hosting company this way.

What I would recommend when choosing a hosting company (or other online-services company) to which you consider giving your money:

check if the company has accreditations from respectable review sources (if the company is based in the USA, look for the BSA for example). Always click trough to the review website and check the details, because some unscrupulous sites display the logos, even though they don’t have any actual accreditations
look up the domain name with domaintools. See how long the domain has been registered (the longer, the better). See the address of the owner. If it is “privacy protected”, it should raise some red flags (while I admit that such services have value for individuals, I would be weary of doing business with companies unwilling to disclose their physical location.
Do a search for “company name scam”, “company name complaints” and other negative terms which come in mind. Do the same with their phone number. While you should be cautious in taking information found on the internet at face value, the prevalence of such articles should make you think twice.

While adhering to these rules will not entirely eliminate the possibility of you being scammed, it should reduce it considerably. I also try to pre-filter any site I write a review about using the above methods.

Full disclosure: this is a paid review from ReviewMe. Under the terms of the understanding I was not obligated to skew my viewpoint in any way (ie. only post positive facts).

eDirectHost review

Attila-Mihaly Balazs — Mon, 22 Mar 2010 10:33:00 +0200

eDirectHost is one of the many companies which offer websites for people who don’t want to get bogged down in technical details. The particular niche eDirectHost focuses on is ecommerce web design. In particular, they offer many features specific to such sites like: coupons, support for different payment providers, etc. Two other positive aspects are that it is BBB accredited (which should give you some confidence in the way they conduct business) and their pricing is competitive with other solutions like Yahoo! Store.

You are giving up some control when going with such solutions, no question about it, so you should carefully evaluate the tradeoff. One recommendation of mine would be to register the domain name separately (ie. not trough them). This way you retain some amount of control and if you have any problems with them, you can set up a site with an other provider and redirect your domain there.

An other concern might be that they are billing you for services which you could get otherwise for free (ie. website analytics, email, etc), however they seem to be reasonable in this department also (the services which you could get for free are included in all the packages).

Finally, you might wonder if the fact that the number of templates is limited is a disadvantage. Most of the people however, when they are shopping online, don’t mind it and are more concentrated on getting the right product for the right price than how the site looks. The only time looks influences their decision is when they estimate the credibility of the site, and professionally designed templates certainly help there.

Full disclosure: this is a paid review from ReviewMe. Under the terms of the understanding I was not obligated to skew my viewpoint in any way (ie. only post positive facts).

Unshortifying Cisco “Go” links

Attila-Mihaly Balazs — Fri, 19 Mar 2010 17:23:00 +0200

Inspired by a post on the PacketLife.net blog - Cisco “Go” links reference in the wiki – I tried to mine the short links to come up with the “definitive” list, but after running it for a couple of days, it only managed to find 473 links, compared to the 4720 Google estimates it has (Yahoo estimates 4200, but it seems to include more non-relevant results – or maybe I don’t know to use the search operators on it correctly :-P). Anyway, I thought that the code might be useful for somebody doing other scraping projects, so you can find it in my Google Code SVN repository. It illustrates a couple of techniques:

Generating all the combinations from a given alphabet in a simple and fast manner
Using multi-threading to increase performance and reduce the time wasted by waiting for network I/O.
How to fetch gzip-ed content (any well-behaved spider should offer the site the option to do so to conserve their bandwidth) using LWP::UserAgent (I found it thanks to this StackOverflow question)

Check it out if you have a similar problem!

Finally, below you have the list of links. A quick look reveals two interesting observations: there are duplicates (multiple links pointing to the same page) and some of the links point to non-Cisco pages.

http://cisco.com/go/f => Redirect page - Cisco Systems(http://www.cisco.com/web/mobile/fed/)
http://cisco.com/go/m => Cisco Systems - Cisco Speaking Sessions at Mobile World Congress (http://www.cisco.com/web/learning/le21/le34/MWC/2009/mobi/)
http://cisco.com/go/n => Cisco Systems - Innovators (http://www.cisco.com/web/mobile/nws/)
http://cisco.com/go/s => Cisco Systems - Test Your Cisco Smarts (http://www.cisco.com/web/mobile/s/)
http://cisco.com/go/ea => Cisco Unified Expert Advisor - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps9675/index.html)
http://cisco.com/go/sa => Cisco IOS Software Activation - Cisco Systems (http://www.cisco.com/en/US/products/ps9677/products_ios_technology_home.html)
http://cisco.com/go/qb => Cisco Systems(http://www.cisco.com/web/partners/quotebuilder)
http://cisco.com/go/ac => Cisco Systems - Unified Attendant Console solutions (http://cisco-ac.arcsolutions.com)
http://cisco.com/go/cc => Customer Contact - Cisco Systems (http://www.cisco.com/en/US/products/sw/custcosw/Products_Sub_Category_Home.html)
http://cisco.com/go/bc => SA Europe - Broadcasters/Programmers (http://www.saeurope.com/solutions/broadcasters.htm)
http://cisco.com/go/dc => Data Center - Cisco Systems (http://www.cisco.com/en/US/netsol/ns340/ns394/ns224/index.html)
http://cisco.com/go/pc => Positive Connections - Operational Excellence through Connected Manufacturing, 19 May 2009, 9am-12pm GMT Webcast (http://www.cisco.com/web/offer/emea/positiveconnections/index.html)
http://cisco.com/go/uc => Voice & Unified Communications - Main Page - Cisco Systems (http://www.cisco.com/en/US/products/sw/voicesw/index.html)
http://cisco.com/go/vc => VoiceCon 2010 - Cisco Events - Cisco Systems(http://www.cisco.com/web/learning/le21/le34/voicecon/2010/index.html)
http://cisco.com/go/id => Cisco Systems, Inc (http://www.cisco.com/web/ID/index.html)
http://cisco.com/go/ce => Carrier Ethernet - Cisco Systems (http://www.cisco.com/en/US/netsol/ns577/networking_solutions_solution.html)
http://cisco.com/go/ie => Cisco Catalyst 2955 Series Switches - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps6738/index.html)
http://cisco.com/go/3g => Cisco 3G Wireless Connectivity Solutions [Cisco 800 Series Routers] - Cisco Systems (http://www.cisco.com/en/US/prod/routers/ps380/3g_solns.html)
http://cisco.com/go/ph => Partner Helpline - Partner Central - Cisco Systems(http://www.cisco.com/web/partners/tools/helponline/index.html)
http://cisco.com/go/dm => Order Direct From Cisco - Cisco Systems (http://www.cisco.com/commarch/cvs/dm)
http://cisco.com/go/sm => Cisco Systems - Redirect to (http://www.cisco.com/humannetwork)
http://cisco.com/go/hn => The Human Network - Cisco Systems (http://www.cisco.com/web/thehumannetwork/index1.html?POSITION=link&COUNTRY;_SITE=us&CAMPAIGN;=HN2&CREATIVE;=HN2+to+HN1&REFERRING;_SITE=CISCO%2ECOM+HN2+Microsite)
http://cisco.com/go/fn => Cisco Feature Navigator - Cisco Systems (http://tools.cisco.com/ITDIT/CFN/jsp/index.jsp)
http://cisco.com/go/sn => Cisco.com Login Page (http://tools.cisco.com/Support/CPI/index.do)
http://cisco.com/go/so => Cisco Learning Partner Associate - Learning Partners Program Overview - Cisco Systems(http://www.cisco.com/web/learning/le27/le53/learning_partner_so.html)
http://cisco.com/go/cp => Cisco Powered Program - Cisco Systems (http://www.cisco.com/en/US/netsol/ns206/networking_solutions_solution_category.html)
http://cisco.com/go/bp => Cisco Systems(http://www.cisco.com/web/partners/program/other/brand-protection/index.html)
http://cisco.com/go/ds => Cisco Digital Signs - Cisco Systems(http://www.cisco.com/web/solutions/dms/digital_signage.html)
http://cisco.com/go/ps => Government and Education - Cisco Systems(http://www.cisco.com/web/strategy/government_education_index.html)
http://cisco.com/go/ts => Technical Services - Cisco Systems (http://www.cisco.com/en/US/products/svcs/ps3034/ps2827/serv_category_home.html)
http://cisco.com/go/tv => Telepresence - Overview - Cisco Systems (http://www.cisco.com/en/US/netsol/ns669/networking_solutions_solution_segment_home.html)
http://cisco.com/go/x2 => Cisco Systems - Redirect to (http://www.cisco.com/en/US/products/hw/modules/ps5455/products_data_sheet0900aecd801f92aa.html)
http://cisco.com/go/24 => President Taylor Meets Over TelePresence On 24 - Video Detail - The Video Lounge (http://videolounge.cisco.com/video/24-pres-taylor-meets-over-tp/?Referring_site=PrintTv&Country;_Site=US&Campaign;=HN&Position;=URL&Creative;=go/24&Where;=go/24)
http://cisco.com/go/saa => Cisco IOS IP Service Level Agreements (SLAs) - Cisco Systems (http://www.cisco.com/en/US/products/ps6602/products_ios_protocol_group_home.html)
http://cisco.com/go/cca => Cisco NAC Appliance (Clean Access) - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps6128/index.html)
http://cisco.com/go/pda => Cisco Systems, Inc (http://www.cisco.com/cdc_content_elements/mobile/)
http://cisco.com/go/sea => The Page You Have Requested Is Not Available (http://www.cisco.com/en/US/strategy/transportation/seaports.html)
http://cisco.com/go/cia => Collaboration In Action (http://www.cisco.com/web/offer/emea/collaborationinaction)
http://cisco.com/go/via => The Page You Have Requested Is Not Available (http://www.cisco.com/en/US/netsol/ns341/ns396/ns166/ns68/networking_solutions_solution.html)
http://cisco.com/go/ana => Cisco Active Network Abstraction - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps6776/index.html)
http://cisco.com/go/cna => Cisco Network Assistant - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps5931/index.html)
http://cisco.com/go/cpa => Cisco Channel Port Adapter - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/hw/modules/ps2033/ps124/index.html)
http://cisco.com/go/lpa => Cisco Learning Partner Associate - Learning Partners Program Overview - Cisco Systems(http://www.cisco.com/web/learning/le27/le53/learning_partner_so.html)
http://cisco.com/go/spa => Cisco Shared Port Adapters/SPA Interface Processors - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps6267/prod_module_series_home.html)
http://cisco.com/go/cqa => Cisco Systems(http://www.cisco.com/web/partners/sell/technology/quoteadvisor.html)
http://cisco.com/go/asa => Cisco ASA 5500 Series Adaptive Security Appliances - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps6120/index.html)
http://cisco.com/go/csa => Cisco Security Agent - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/sw/secursw/ps5057/index.html)
http://cisco.com/go/isa => CCO Decommission Page (http://www.cisco.com/warp/public/732/Tech/connectivity/ssg/)
http://cisco.com/go/msa => Introduction - Cisco Systems(http://www.cisco.com/en/US/partners/pr67/pr41/pr263/partners_strategic_solution_concept_home.html)
http://cisco.com/go/vsa => Cisco VPN Services Adapter - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps7332/index.html)
http://cisco.com/go/cta => NAC - Cisco Systems (http://www.cisco.com/en/US/netsol/ns466/networking_solutions_package.html)
http://cisco.com/go/aya => Are You Attached Seminar Series - 1 Day (https://programs.regweb.com/cisco/aya/)
http://cisco.com/go/cab => Cisco.com Login Page (http://forums.cisco.com/eforum/servlet/CAB?page=main&sn;=CAB)
http://cisco.com/go/mib => Cisco IOS MIB Locator (http://tools.cisco.com/ITDIT/MIBS/servlet/index)
http://cisco.com/go/sib => Small is BIG (http://www.cisco.com/web/EA/sib/index.html)
http://cisco.com/go/brb => Branch - Cisco Systems (http://www.cisco.com/en/US/netsol/ns477/index.html)
http://cisco.com/go/irb => Thought Leadership Network (http://newsroom.cisco.com/dlls/tln/redirects/irb_metric.html)
http://cisco.com/go/fsb => The Page You Have Requested Is Not Available (http://www.cisco.com/en/US/netsol/ns477/networking_solutions_packages_list.html)
http://cisco.com/go/hsb => Cisco Hosted Small Business Communications - Cisco Systems (http://www.cisco.com/en/US/netsol/ns1028/networking_solutions_solution.html)
http://cisco.com/go/twb => Teachers Without Borders (http://www.cisco.com/web/learning/netacad/landing/TWB.html)
http://cisco.com/go/syb => Cisco Systems(http://www.cisco.com/web/partners/sell/technology/security/secure_your_branch.html)
http://cisco.com/go/dac => Cisco Unified Department Attendant Console - Cisco Systems (http://www.cisco.com/en/US/products/ps7295/index.html)
http://cisco.com/go/eac => Cisco Physical Access Gateways - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps9687/index.html)
http://cisco.com/go/nac => NAC - Cisco Systems (http://www.cisco.com/en/US/netsol/ns466/networking_solutions_package.html)
http://cisco.com/go/ibc => Join Us at IBC 2009 (http://www.scientificatlanta.com/email/2009/0709-IBC/new/JoinCiscoatIBC2009.htm)
http://cisco.com/go/nbc => 30 Rock/Jenna Finds A Flip - Video Detail - The Video Lounge (http://videolounge.cisco.com/video/30-rockjenna-finds-a-flip/?Referring_site=PrintTV&Country;_Site=US&Campaign;=Product+Integrations&Position;=Vanity&Creative;=http://www.cisco.com/go/nbc)
http://cisco.com/go/sbc => The Page You Have Requested Is Not Available (http://www.cisco.com/en/US/netsol/ns759/networking_solutions_sub_sub_solution.html)
http://cisco.com/go/pec => Partner Education Connection - Training Resources - Cisco Systems(http://www.cisco.com/web/learning/le36/learning_partner_e-learning_connection_tool_launch.html)
http://cisco.com/go/cfc => Order Direct From Cisco - Cisco Systems (http://www.cisco.com/commarch/cvs/cfc)
http://cisco.com/go/dfc => Log-In (http://resources.cisco.com/app/channel-site-builder.taf?channel_id=32631&public;_view=true&asset;_id=5926)
http://cisco.com/go/cic => Cisco Info Center - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/sw/netmgtsw/ps996/index.html)
http://cisco.com/go/tlc => Federal IT Thought Leadership - Industry Solutions - Cisco Systems(http://www.cisco.com/web/strategy/government/federal_thought_leadership.html)
http://cisco.com/go/bmc => DC Partner - BMC - Cisco Systems (http://www.cisco.com/en/US/netsol/ns957/index.html)
http://cisco.com/go/pmc => Cisco Systems(http://www.cisco.com/web/partners/services/resources/pmc/index.html)
http://cisco.com/go/anc => Inventory and Reporting - Cisco Systems (http://www.cisco.com/kobayashi/support/home.htm)
http://cisco.com/go/voc => Cisco.com Login Page (http://www.cisco.com/en/US/customer/ordering/o44/ordering_concept_home.html)
http://cisco.com/go/epc => Cisco IOS Embedded Packet Capture - Cisco Systems (http://www.cisco.com/en/US/products/ps9913/products_ios_protocol_group_home.html)
http://cisco.com/go/ipc => Unified Communications/Voice Solutions - Cisco Systems (http://www.cisco.com/en/US/netsol/ns340/ns394/ns165/networking_solutions_packages_list.html)
http://cisco.com/go/hpc => The Page You Have Requested Is Not Available (http://www.cisco.com/en/US/netsol/ns500/networking_solutions_package.html)
http://cisco.com/go/rrc => Financial Services - Industry Solutions - Cisco Systems(http://www.cisco.com/web/strategy/financial/index.html)
http://cisco.com/go/isc => Cisco IP Solution Center - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/sw/netmgtsw/ps4748/index.html)
http://cisco.com/go/psc => Partner Help Online-Partners & Resellers - Cisco Systems (http://ciscopsc.custhelp.com/cgi-bin/ciscopsc.cfg/php/enduser/cisco.php)
http://cisco.com/go/ssc => Partner Help Online-Partners & Resellers - Cisco Systems (http://ciscopsc.custhelp.com/cgi-bin/ciscopsc.cfg/php/enduser/cisco.php)
http://cisco.com/go/cuc => The Page You Have Requested Is Not Available (http://www.cisco.com/en/US/netsol/ns641/networking_solutions_packages_list.html)
http://cisco.com/go/mwc => Cisco Mobile Wireless Center - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/sw/netmgtsw/ps820/index.html)
http://cisco.com/go/iad => Cisco IAD2400 Series Integrated Access Devices - Cisco Systems (http://www.cisco.com/en/US/products/hw/gatecont/ps887/index.html)
http://cisco.com/go/fed => e-government - U.S. Federal Government -Industry Solutions - Cisco Systems(http://www.cisco.com/web/strategy/government/us_federal.html)
http://cisco.com/go/cmd => Cisco Monitor Director - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps7246/index.html)
http://cisco.com/go/usd => Cisco Unified Service Delivery (http://www.cisco.com/cdc_content_elements/flash/netsol/sp/sdc/index.html?POSITION=printvanity&COUNTRY;_SITE=us&CAMPAIGN;=SDC&CREATIVE;=Vanity&REFERRING;_SITE=Vanity+URL)
http://cisco.com/go/cvd => Cisco Validated Design Program - Cisco Systems (http://www.cisco.com/en/US/netsol/ns741/networking_solutions_program_home.html)
http://cisco.com/go/ace => Data Center Application Services - Cisco Systems (http://www.cisco.com/en/US/products/ps5719/Products_Sub_Category_Home.html)
http://cisco.com/go/dce => The Page You Have Requested Is Not Available (http://www.cisco.com/en/US/netsol/ns783/networking_solutions_package.html#\~overview)
http://cisco.com/go/nce => Cisco Network Capacity Expansion - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps9702/index.html)
http://cisco.com/go/tce => News@Cisco -> Executive Biographies (http://tools.cisco.com/dlls/tln/page/business/biz-customer-experience)
http://cisco.com/go/mde => Cisco MPLS Diagnostics Expert - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps6755/index.html)
http://cisco.com/go/ime => Cisco IPS Manager Express - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps9610/index.html)
http://cisco.com/go/mme => The Page You Have Requested Is Not Available (http://www.cisco.com/web/partners/pr47/pr288/partners_marketing_made_easy.html)
http://cisco.com/go/ppe => Cisco.com Login Page (http://tools.cisco.com/WWChannels/PPP/home.do?actionType=home)
http://cisco.com/go/ase => Redirect (http://www.cisco.com/warp/public/437/services/ndm/aes.html)
http://cisco.com/go/ese => Enterprise - Cisco Systems (http://www.cisco.com/warp/public/779/largeent/it/ese/srnd.html)
http://cisco.com/go/cse => Cisco Solutions Express (http://www.cisco.com/cdc_content_elements/flash/large_enterprise/truck.html)
http://cisco.com/go/mse => Cisco 3300 Series Mobility Services Engine - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps9742/index.html)
http://cisco.com/go/cue => Cisco Unity Express - Cisco Systems (http://www.cisco.com/en/US/products/sw/voicesw/ps5520/index.html)
http://cisco.com/go/waf => Cisco ACE Web Application Firewall - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps9586/index.html)
http://cisco.com/go/ccf => Network Fabric - Cisco Systems (http://www.cisco.com/en/US/netsol/ns725/index.html)
http://cisco.com/go/pdf => Cisco Systems(http://www.cisco.com/web/partners/sell/smb/programs_and_promotions/pdf.html)
http://cisco.com/go/sef => Service Exchange Framework - Cisco Systems (http://www.cisco.com/en/US/netsol/ns746/networking_solutions_sub_solution.html)
http://cisco.com/go/jmf => We Apologize - 401 Error(http://www.cisco.com/cgi-bin/front.x/jmf/jmf30/jmf30/SelectCountry)
http://cisco.com/go/fnf => Flexible NetFlow - Cisco Systems (http://www.cisco.com/en/US/products/ps6965/products_ios_protocol_option_home.html)
http://cisco.com/go/crf => Cisco.com Login Page (https://tools.cisco.com/WWChannels/MBO/SMB/home.do)
http://cisco.com/go/gsf => 2010 Government Solutions Forum (http://www.cisco.com/web/strategy/government/solutionsforum.html)
http://cisco.com/go/atf => Cisco Systems: Business Discussion Forum - Login (http://forums.cisco.com/eforum/servlet/CBDF?page=cbdf&forum;=CBDF%20Forum&topic;=Event%20Discussions&CommCmd;=MB%3Fcmd%3Ddisplay_location%26location%3D.2cc298a5)
http://cisco.com/go/ctf => The Page You Have Requested Is Not Available (http://www.cisco.com/web/strategy/government/Public_Sector_Technology_Forum.html)
http://cisco.com/go/gtf => News@Cisco -> Executive Biographies (http://newsroom.cisco.com/dlls/tln/events/gtf/index.html)
http://cisco.com/go/tmg => Cisco Transceiver Modules - Support - Cisco Systems (http://www.cisco.com/en/US/products/hw/modules/ps5455/tsd_products_support_series_home.html)
http://cisco.com/go/qrg => Cisco Product Quick Reference Guide - Cisco Systems (http://www.cisco.com/en/US/prod/qrg/index.html)
http://cisco.com/go/isg => Cisco Intelligent Services Gateway - Cisco Systems (http://www.cisco.com/en/US/products/ps6588/products_ios_protocol_group_home.html)
http://cisco.com/go/ssg => Service Selection Gateway - Cisco Systems (http://www.cisco.com/en/US/products/ps6589/products_ios_protocol_group_home.html)
http://cisco.com/go/cug => Cisco Community Central: Community: Cisco User Groups (https://www.myciscocommunity.com/community/technology/collaboration/usergroups)
http://cisco.com/go/pbi => Support for Nonprofits - Cisco Systems(http://www.cisco.com/web/about/ac48/pbi.html)
http://cisco.com/go/dci => Data Center Interconnect - Cisco Systems (http://www.cisco.com/en/US/netsol/ns975/index.html)
http://cisco.com/go/pci => PCI Compliance - Cisco Systems (http://www.cisco.com/en/US/netsol/ns625/index.html)
http://cisco.com/go/udi => Products & Services Product Identification Standard - Cisco Systems (http://www.cisco.com/en/US/products/products_identification_standard.html)
http://cisco.com/go/aii => Projects - Cisco Systems(http://www.cisco.com/en/US/about/ac50/ac207/projects/index.html)
http://cisco.com/go/vni => Visual Networking Index - Cisco Systems (http://www.cisco.com/en/US/netsol/ns827/networking_solutions_sub_solution.html)
http://cisco.com/go/cpi => Cisco Systems(http://www.cisco.com/web/partners/news/index.html)
http://cisco.com/go/ppi => Cisco.com Login Page (https://apps.cisco.com/mbrprv/saw.dll?Dashboard)
http://cisco.com/go/vpi => Cisco Systems(http://www.cisco.com/web/partners/pr46/vpi/vpi.html)
http://cisco.com/go/ipj => The Internet Protocol Journal - ISSN 1944-1134 - Cisco Systems(http://www.cisco.com/en/US/about/ac123/ac147/about_cisco_the_internet_protocol_journal.html)
http://cisco.com/go/ask => Cisco Support Community: Cisco Support Community (http://forum.cisco.com/eforum/servlet/NetProf?page=main)
http://cisco.com/go/pal => Cisco.com Login Page (http://tools.cisco.com/WWChannels/PAL/index.jsp)
http://cisco.com/go/ccl => Cisco Collaboration - Introduction - Cisco Systems(http://www.cisco.com/web/partners/sell/technology/ipc/cisco_collaboration.html)
http://cisco.com/go/oil => Oil and Gas - Industry Solutions - Cisco Systems(http://www.cisco.com/web/strategy/energy/external_oil.html)
http://cisco.com/go/etl => News@Cisco -> Executive Biographies (http://newsroom.cisco.com/dlls/tln/)
http://cisco.com/go/nam => Network Analysis Module (NAM) Products - Cisco Systems (http://www.cisco.com/en/US/products/ps5740/Products_Sub_Category_Home.html)
http://cisco.com/go/vam => Log-In (http://resources.cisco.com/app/tree.taf?asset_id=57360&public;_view=true)
http://cisco.com/go/fbm => Forbidden File or Application(http://www.cisco.com/web/strategy/financial/index.html)
http://cisco.com/go/ndm => Introduction - Advanced Services Education - Cisco Systems(http://www.cisco.com/web/learning/le31/ase/index.html)
http://cisco.com/go/pdm => Cisco PIX Device Manager - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/sw/netmgtsw/ps2032/index.html)
http://cisco.com/go/eem => Cisco IOS Embedded Event Manager (EEM) - Cisco Systems (http://www.cisco.com/en/US/products/ps6815/products_ios_protocol_group_home.html)
http://cisco.com/go/eim => Cisco Unified E-Mail Interaction Manager - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps7236/index.html)
http://cisco.com/go/pim => IP Multicast - Cisco Systems (http://www.cisco.com/en/US/products/ps6552/products_ios_technology_home.html)
http://cisco.com/go/rim => The Page You Have Requested Is Not Available (http://www.cisco.com/en/US/prod/collateral/voicesw/product_promotion0900aec806e252a.html)
http://cisco.com/go/wim => Cisco Unified Web Interaction Manager - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps7233/index.html)
http://cisco.com/go/clm => Cisco License Manager - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps7138/index.html)
http://cisco.com/go/cmm => Cisco Multicast Manager - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps6337/index.html)
http://cisco.com/go/anm => Cisco Application Networking Manager - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps6904/index.html)
http://cisco.com/go/enm => Network Management - Main Page - Cisco Systems (http://www.cisco.com/en/US/products/sw/netmgtsw/index.html)
http://cisco.com/go/com => Order Direct From Cisco - Cisco Systems (http://www.cisco.com/commarch/cvs/com)
http://cisco.com/go/fpm => Cisco IOS Flexible Packet Matching (FPM) - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps6723/index.html)
http://cisco.com/go/epm => Cisco Policy Management - Cisco Systems (http://www.cisco.com/en/US/products/ps9519/Products_Sub_Category_Home.html)
http://cisco.com/go/bqm => Cisco Bandwidth Quality Manager - Network Planning - Products & Services - Cisco Systems - Cisco Systems (http://www.cisco.com/en/US/products/ps6385/index.html)
http://cisco.com/go/asm => Cisco AnyConnect Secure Mobility Solution - Cisco Systems (http://www.cisco.com/en/US/netsol/ns1049/index.html)
http://cisco.com/go/lsm => IP Multicast - Cisco Systems (http://www.cisco.com/en/US/products/ps6552/products_ios_technology_home.html)
http://cisco.com/go/ctm => Cisco Transport Manager - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/sw/opticsw/ps2204/index.html)
http://cisco.com/go/hum => CiscoWorks Health and Utilization Monitor - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps9303/index.html)
http://cisco.com/go/pvm => Cisco Performance Visibility Manager - Network Performance - Products & Services - Cisco Systems - Cisco Systems (http://www.cisco.com/en/US/products/ps6768/index.html)
http://cisco.com/go/san => The Page You Have Requested Is Not Available (http://www.cisco.com/en/US/netsol/ns893/networking_solutions_package.html)
http://cisco.com/go/wan => Unified WAN Services Platforms [Routers] - Cisco Systems (http://www.cisco.com/en/US/prod/routers/networking_solutions_products_unified_wan_services_platforms.html)
http://cisco.com/go/sbn => Cisco Systems(http://www.cisco.com/web/partners/sell/technology/security/borderless_security.html)
http://cisco.com/go/cdn => Cisco Developer Community - Home - Cisco Developer Community (http://developer.cisco.com)
http://cisco.com/go/sdn => Security Solutions for Enterprise - Cisco Systems (http://www.cisco.com/en/US/netsol/ns340/ns394/ns171/networking_solutions_packages_list.html)
http://cisco.com/go/cfn => Cisco Feature Navigator - Cisco Systems (http://tools.cisco.com/ITDIT/CFN/)
http://cisco.com/go/cin => The Page You Have Requested Is Not Available (http://www.cisco.com/web/partners/events/cin.html)
http://cisco.com/go/aon => Application-Oriented Networking - Cisco Systems (http://www.cisco.com/en/US/products/ps6692/Products_Sub_Category_Home.html)
http://cisco.com/go/cpn => Shortcut Redirect - Cisco Systems (http://www.cisco.com/pcgi-bin/cpn/cpn_pub_bassrch.pl)
http://cisco.com/go/vpn => Virtual Private Networks (VPN) - Main Page - Cisco Systems (http://www.cisco.com/en/US/products/ps5743/Products_Sub_Category_Home.html)
http://cisco.com/go/brn => Places in the Network - Cisco Systems (http://www.cisco.com/en/US/netsol/ns936/index.html)
http://cisco.com/go/cio => CIO - Cisco Systems (http://www.cisco.com/en/US/netsol/ns1018/index.html)
http://cisco.com/go/sio => Security Intelligence Operations - Cisco Systems (http://tools.cisco.com/security/center/home.x)
http://cisco.com/go/cpo => (https://tools.cisco.com/gdrp/coiga/showsurvey.do?surveyCode=445&keyCode;=106721_1)
http://cisco.com/go/map => 401 Authorization Required (http://www.cisco-services.com/map)
http://cisco.com/go/sap => DC Partner - SAP - Cisco Systems (http://www.cisco.com/en/US/netsol/ns970/index.html)
http://cisco.com/go/tap => Cisco Systems(http://www.cisco.com/web/partners/pr11/incentive/tap/index.html)
http://cisco.com/go/tbp => Cisco Trusted Business Professional (https://programs.regweb.com/cisco/ctbp_08/)
http://cisco.com/go/ccp => Cisco Configuration Professional - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps9422/index.html)
http://cisco.com/go/gep => Global EasyPay (GEP) - Partner Central - Cisco Systems(http://www.cisco.com/web/partners/tools/global_easypay.html)
http://cisco.com/go/rep => Cisco.com Login Page (http://tools.cisco.com/WWChannels/CAMLOC/jsp/cam_locator.jsp)
http://cisco.com/go/nfp => Cisco IOS Network Foundation Protection (NFP) - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps6642/index.html)
http://cisco.com/go/agp => Cisco.com Login Page (https://tools.cisco.com/WWChannels/MBO/home.do)
http://cisco.com/go/cip => Cisco Channel Interface Processors - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/hw/modules/ps2643/ps123/index.html)
http://cisco.com/go/dip => We Apologize - 401 Error(http://www.cisco.com/global/EMEA/pages/dip/)
http://cisco.com/go/sip => Cisco Systems(http://www.cisco.com/web/partners/pr11/incentive/sip.html)
http://cisco.com/go/vip => Cisco.com Login Page (http://www.cisco.com/en/US/partner/partners/pr11/incentive/vip.shtml)
http://cisco.com/go/oip => Cisco.com Login Page (http://www.cisco.com/en/US/partner/partners/pr11/incentive/oip.shtml)
http://cisco.com/go/sjp => Cisco Systems(http://www.cisco.com/web/partners/pr11/incentive/sip.html)
http://cisco.com/go/dlp => Data Loss Prevention - Cisco Systems (http://cisco.com/en/US/netsol/ns895/index.html)
http://cisco.com/go/clp => Cisco Learning Partner - Learning Partners Program Overview - Cisco Systems(http://www.cisco.com/web/learning/le27/le53/learning_partner_clp.html)
http://cisco.com/go/dmp => Cisco.com Login Page (http://tools.cisco.com/emea/dmt/index.jsp)
http://cisco.com/go/cpp => Cisco Powered Program for the Service Provider - Cisco Systems (http://www.cisco.com/en/US/netsol/ns851/networking_solutions_solution.html)
http://cisco.com/go/dpp => Cisco.com Login Page (https://tools.cisco.com/WWChannels/MBO/home.do)
http://cisco.com/go/app => Other Cisco Programs - Partner Central - Cisco Systems(http://www.cisco.com/en/US/partners/pr46/partners_pgm_category_page.html)
http://cisco.com/go/urp => Redirect Notification - This page has moved to a new location! - Cisco Systems(http://www.cisco.com/web/about/ac50/ac207/crc/index1.html)
http://cisco.com/go/asp => Log-In (http://resources.cisco.com/app/tree.taf?asset_id=482157&public;_view=true)
http://cisco.com/go/ksp => Knowledge Services - Advanced Services Education - Cisco Systems(http://www.cisco.com/web/learning/le31/ase/knowledgeservices/index.html)
http://cisco.com/go/psp => Public Sector Program (PSP) - Cisco Systems(http://www.cisco.com/web/partners/pr11/incentive/em/em_psp.html)
http://cisco.com/go/atp => Authorized Technology Provider (ATP) Program - Partner Central - Cisco Systems(http://www.cisco.com/web/partners/pr11/atp/index.html)
http://cisco.com/go/cvp => Cisco Unified Customer Voice Portal - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/sw/custcosw/ps1006/index.html)
http://cisco.com/go/axp => Optimize Branch Footprint with Application Integration [Cisco Application Extension Platform] - Cisco Systems (http://www.cisco.com/en/US/prod/routers/ps9701/axp_promo.html)
http://cisco.com/go/p4p => Cisco Systems(http://www.cisco.com/web/partners/services/promos/p4p/index.html)
http://cisco.com/go/sbr => Cisco Systems(http://www.cisco.com/web/partners/sell/smb/tools_and_resources/smart_business_roadmap.html)
http://cisco.com/go/pdr => Cisco.com Login Page (https://cisco-apps.cisco.com/cisco/psn/commerce)
http://cisco.com/go/oer => Cisco Optimized Edge Routing - Cisco Systems (http://www.cisco.com/en/US/products/ps6628/products_ios_protocol_option_home.html)
http://cisco.com/go/pfr => Performance Routing - Cisco Systems (http://www.cisco.com/en/US/products/ps8787/products_ios_protocol_option_home.html)
http://cisco.com/go/ehr => 502 Proxy Error (http://www.cisco.com/web/strategy/healthcare/breathe_life_into_ehr.html)
http://cisco.com/go/air => Airports - Transportation - Cisco Systems(http://www.cisco.com/en/US/strategy/transportation/airports.html)
http://cisco.com/go/asr => Introducing the Cisco ASR 1000 Router Series [Cisco ASR 1000 Series Aggregation Services Routers] - Cisco Systems (http://www.cisco.com/en/US/prod/routers/ps9343/asr_1000_prod_announcement.html)
http://cisco.com/go/abs => (http://www.crmtool.net/WebForm.asp?F=251&W;=2824)
http://cisco.com/go/cbs => WebEx Helps Crack The Case On CSI: Miami - Video Detail - The Video Lounge (http://videolounge.cisco.com/video/csi-miami-webex-cracks-case/?Referring_site=PrintTv&Country;_Site=US&Campaign;=HN&Position;=URL&Creative;=go/cbs&Where;=go/cbs)
http://cisco.com/go/acs => Cisco Secure Access Control System - Cisco Systems (http://cisco.com/en/US/products/ps9911/index.html)
http://cisco.com/go/ics => Cisco Incident Control System - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps6542/index.html)
http://cisco.com/go/scs => Secure Remote Access and VPN - Cisco Systems (http://www.cisco.com/en/US/netsol/ns461/networking_solutions_package.html)
http://cisco.com/go/cds => Content Delivery Systems - Cisco Systems (http://www.cisco.com/en/US/products/ps7191/Products_Sub_Category_Home.html)
http://cisco.com/go/ids => Shortcut Redirect - Cisco Systems (http://www.cisco.com/go/ips)
http://cisco.com/go/tds => Threat Control - Cisco Systems (http://www.cisco.com/en/US/netsol/ns340/ns394/ns171/ns441/networking_solutions_package.html)
http://cisco.com/go/ams => Cisco Assurance Management Solution - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps8408/index.html)
http://cisco.com/go/dms => Digital Media Suite - Cisco Systems(http://www.cisco.com/web/solutions/dms/)
http://cisco.com/go/nms => Network Management - Main Page - Cisco Systems (http://www.cisco.com/en/US/products/sw/netmgtsw/index.html)
http://cisco.com/go/sms => Text Messaging at Cisco - About Cisco - Cisco Systems(http://www.cisco.com/web/about/facts_info/sms_reg_info.html)
http://cisco.com/go/ans => Application Networking Services - Main Page - Cisco Systems (http://www.cisco.com/en/US/products/hw/contnetw/index.html)
http://cisco.com/go/dns => Dynamic Host Control Protocol (DHCP)/Domain Name System (DNS) - Cisco Systems (http://www.cisco.com/en/US/products/ps6641/products_ios_protocol_option_home.html)
http://cisco.com/go/eos => Cisco Certified Refurbished Equipment - Cisco Capital Finance - Cisco Systems(http://www.cisco.com/web/ordering/ciscocapital/refurbished/index.html)
http://cisco.com/go/ios => Cisco IOS and NX-OS Software - Main Page - Cisco Systems (http://www.cisco.com/en/US/products/sw/iosswrel/products_ios_cisco_ios_software_category_home.html)
http://cisco.com/go/qos => Quality of Service (QoS) - Cisco Systems (http://www.cisco.com/en/US/products/ps6558/products_ios_technology_home.html)
http://cisco.com/go/ros => Remote Management Services - Cisco Systems (http://www.cisco.com/en/US/products/ps6192/serv_category_home.html)
http://cisco.com/go/wos => Overview - Exhibit & Sponsorship Opportunities - Cisco Systems(http://www.cisco.com/web/learning/le21/le34/networkers/nw07/wos)
http://cisco.com/go/ips => Cisco Intrusion Prevention System - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/sw/secursw/ps2113/index.html)
http://cisco.com/go/ops => Cisco.com Login Page (http://www.cisco.com/cgi-bin/cpn/show_page.pl?file_name=symposiums.html&type;=technical)
http://cisco.com/go/crs => Cisco Carrier Routing System - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps5763/index.html)
http://cisco.com/go/oss => Network Management - Main Page - Cisco Systems (http://www.cisco.com/en/US/products/sw/netmgtsw/index.html)
http://cisco.com/go/pss => Cisco.com Login Page (http://tools.cisco.com/WWChannels/GETLOG/welcome.do)
http://cisco.com/go/tss => Technical Services - Cisco Systems (http://www.cisco.com/en/US/products/svcs/ps3034/serv_category_home.html)
http://cisco.com/go/vss => Cisco Catalyst 6500 Virtual Switching System 1440 - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps9336/index.html)
http://cisco.com/go/fts => Cisco Focused Technical Support Services - Cisco Systems (http://www.cisco.com/en/US/products/svcs/ps11/ps2566/ps2567/serv_group_home.html)
http://cisco.com/go/avs => Cisco AVS 3100 Series Application Velocity System - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps6492/index.html)
http://cisco.com/go/uws => Unified WAN Services - Cisco Systems (http://www.cisco.com/en/US/netsol/ns780/index.html)
http://cisco.com/go/act => Cisco Systems, Inc (http://www.cisco.com)
http://cisco.com/go/ect => The Page You Have Requested Is Not Available (http://www.cisco.com/en/US/netsol/ns855/networking_solutions_package.html)
http://cisco.com/go/hft => HFT: Algo Speed - Financial Markets - Financial Markets - Cisco Systems(http://www.cisco.com/web/strategy/financial/algo_speed.html)
http://cisco.com/go/pit => Cisco.com Login Page (http://tools.cisco.com/Support/CPI/index.do)
http://cisco.com/go/int => í+µíc í.Oí_¬ë+_ë¡oì_ - ì`ì+Oê,°ì-. ì+”ë£”ì.\~ - Cisco Systems(http://www.cisco.com/web/KR/networking/smbiz/integrated_tech.html)
http://cisco.com/go/ipt => The Page You Have Requested Is Not Available (http://www.cisco.com/en/US/netsol/ns340/ns394/ns165/ns268/networking_solutions_package.html)
http://cisco.com/go/prt => Partner Relationship Team - Cisco Systems (http://tools.cisco.com/elearning/knet/faq/jsp/faqcontroller.jsp?action=faqList&type;=0:1&module;=FAQ&appid;=11625&rootcatid;=11625&targetID;=11625)
http://cisco.com/go/fst => Financial Services - Industry Solutions - Cisco Systems(http://www.cisco.com/web/strategy/financial/index.html)
http://cisco.com/go/att => Cisco - AT&T; UP - Thank You (https://programs.regweb.com/cisco/stayingoncourse_thankyou/)
http://cisco.com/go/ett => Log-In (http://resources.cisco.com/app/tree.taf?asset_id=126014&public;_view=true&LeftNavID;=142323)
http://cisco.com/go/ftt => Fast Track Trade In (http://tools.cisco.com/WWChannels/MBO/FTT/home.html)
http://cisco.com/go/rtt => Log-In (http://resources.cisco.com/app/tree.taf?asset_id=136369&public;_view=true&randomid;=0.1&LeftNavID;=136369)
http://cisco.com/go/evt => Cisco Systems(http://www.cisco.com/web/partners/sell/technology/ipc/announcements/evt_roadshow.html)
http://cisco.com/go/pvt => Cisco Partner Virtual Team Events (https://programs.regweb.com/cisco/pvt_08/)
http://cisco.com/go/cnu => Cisco Systems(http://www.cisco.com/web/partners/pr46/cnu/index.html)
http://cisco.com/go/gov => e-government - U.S. Federal Government -Industry Solutions - Cisco Systems(http://www.cisco.com/web/strategy/government/us_federal.html)
http://cisco.com/go/cpv => Cisco.com Login Page (http://tools.cisco.com/Support/mytechsupport/index.jsp)
http://cisco.com/go/pgw => Cisco PGW 2200 Softswitch - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/hw/vcallcon/ps2027/index.html)
http://cisco.com/go/fax => Cisco Fax Server - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps6178/index.html)
http://cisco.com/go/ccx => Cisco Compatible Extensions Client Devices - Cisco Compatible Extensions - Cisco Systems(http://www.cisco.com/web/partners/pr46/pr147/partners_pgm_partners_0900aecd800a7907.html)
http://cisco.com/go/fox => President Taylor Meets Over TelePresence On 24 - Video Detail - The Video Lounge (http://videolounge.cisco.com/video/24-pres-taylor-meets-over-tp/?Referring_site=PrintTv&Country;_Site=US&Campaign;=HN&Position;=URL&Creative;=go/fox&Where;=go/fox)
http://cisco.com/go/biz => Internal Server Error (http://www.cisco.com/cisco/web/solutions/small_business/index.html?Referring_site=PrintTv&Country;_Site=us&Campaign;=SAMBA&Position;=Vanity&Creative;=go/biz&Where;=go/biz)
http://cisco.com/go/100 => Cisco SB 100 Series Small-Business Routers - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps6216/index.html)
http://cisco.com/go/850 => Cisco 851 Integrated Services Router - Cisco Systems (http://www.cisco.com/en/US/products/ps6195/index.html)
http://cisco.com/go/360 => > Cisco 360 Learning Program - The Cisco Learning Network (https://learningnetwork.cisco.com/community/learning_center/cisco_360)
http://cisco.com/go/870 => Cisco 871 Integrated Services Router - Cisco Systems (http://www.cisco.com/en/US/products/ps6200/index.html)
http://cisco.com/go/fs1 => Financial Services - Industry Solutions - Cisco Systems(http://www.cisco.com/web/strategy/financial/index.html)
http://cisco.com/go/tv1 => Telepresence - Overview - Cisco Systems (http://www.cisco.com/en/US/netsol/ns669/networking_solutions_solution_segment_home.html)
http://cisco.com/go/vc2 => VoiceCon 2010 - Cisco Events - Cisco Systems(http://www.cisco.com/web/learning/le21/le34/voicecon/2010/index.html)
http://cisco.com/go/fs2 => Financial Services - Industry Solutions - Cisco Systems(http://www.cisco.com/web/strategy/financial/index.html)
http://cisco.com/go/tv2 => Telepresence - Overview - Cisco Systems (http://www.cisco.com/en/US/netsol/ns669/networking_solutions_solution_segment_home.html)
http://cisco.com/go/fs3 => Financial Services - Industry Solutions - Cisco Systems(http://www.cisco.com/web/strategy/financial/index.html)
http://cisco.com/go/tv3 => Telepresence - Overview - Cisco Systems (http://www.cisco.com/en/US/netsol/ns669/networking_solutions_solution_segment_home.html)
http://cisco.com/go/tv4 => Telepresence - Overview - Cisco Systems (http://www.cisco.com/en/US/netsol/ns669/networking_solutions_solution_segment_home.html)
http://cisco.com/go/tv5 => Telepresence - Overview - Cisco Systems (http://www.cisco.com/en/US/netsol/ns669/networking_solutions_solution_segment_home.html)
http://cisco.com/go/tv6 => Telepresence - Overview - Cisco Systems (http://www.cisco.com/en/US/netsol/ns669/networking_solutions_solution_segment_home.html)
http://cisco.com/go/tv7 => Telepresence - Overview - Cisco Systems (http://www.cisco.com/en/US/netsol/ns669/networking_solutions_solution_segment_home.html)
http://cisco.com/go/tv8 => Telepresence - Overview - Cisco Systems (http://www.cisco.com/en/US/netsol/ns669/networking_solutions_solution_segment_home.html)
http://cisco.com/go/tv9 => Telepresence - Overview - Cisco Systems (http://www.cisco.com/en/US/netsol/ns669/networking_solutions_solution_segment_home.html)
http://cisco.com/go/pica => Cisco.com Login Page (http://www.cisco.com/cgi-bin/front.x/pica/welcome_2_pica.pl)
http://cisco.com/go/cpda => The Page You Have Requested Is Not Available (http://www.cisco-powered.com/cp/auth/marketing_sales_resources/cisco_powered_demand_accelerator/)
http://cisco.com/go/ctia => CTIA Wireless 2010 - Cisco Events - Cisco Systems(http://www.cisco.com/web/learning/le21/le34/ctia/2010/index.html)
http://cisco.com/go/nila => News@Cisco -> Executive Biographies (http://newsroom.cisco.com/dlls/tln/research_studies/nila/index.html)
http://cisco.com/go/eula => End User License Agreement [Products & Services] - Cisco Systems (http://www.cisco.com/en/US/docs/general/warranty/English/EU1KEN_.html)
http://cisco.com/go/icpa => Cisco.com Login Page (http://tools.cisco.com/WWChannels/IPA/welcome.do#)
http://cisco.com/go/vspa => Cisco Catalyst 6500 Series VPN Services Port Adapter - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps9893/index.html)
http://cisco.com/go/cspa => Cisco Service Path Analyzer - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps9498/index.html)
http://cisco.com/go/ncta => NCTA Show 2010 - Cisco Events - Cisco Systems(http://www.cisco.com/web/learning/le21/le34/ncta/2010/index.html)
http://cisco.com/go/mfib => IP Multicast - Cisco Systems (http://www.cisco.com/en/US/products/ps6552/products_ios_technology_home.html)
http://cisco.com/go/bnac => Cisco BioMed Network Admission Control - Industry Solutions - Cisco Systems(http://www.cisco.com/web/strategy/healthcare/bioMed_nac.html)
http://cisco.com/go/bpac => Cisco Systems: Business Policy Advisory Council - Login (http://forums.cisco.com/eforum/servlet/BPAC?page=main)
http://cisco.com/go/cabc => Cisco Hardware Inspection and Software Re-Licensing Program [Products & Services] - Cisco Systems (http://www.cisco.com/en/US/prod/hw_sw_relicensing_program.html#\~using)
http://cisco.com/go/brdc => The Page You Have Requested Is Not Available (http://www.cisco.com/en/US/netsol/ns340/ns394/ns224/networking_solutions_packages_list.html)
http://cisco.com/go/trec => Smart+Connected Real Estate - Industry Solutions - Cisco Systems(http://www.cisco.com/web/strategy/trec/index.html)
http://cisco.com/go/psfc => Cisco.com Login Page (http://tools.cisco.com/salesit/psfc/index.jsp)
http://cisco.com/go/gbic => Cisco Transceiver Modules - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/hw/modules/ps5455/prod_module_series_home.html)
http://cisco.com/go/celc => Cisco Learning Home - The Cisco Learning Network (https://cisco.hosted.jivesoftware.com/index.jspa?ciscoHome=true)
http://cisco.com/go/cumc => Cisco Unified Mobile Communicator - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps7271/index.html)
http://cisco.com/go/cepc => Cisco Experience Provider Central (http://ciscoepcentral.veplatform.com)
http://cisco.com/go/hipc => Cisco Powered Program (http://www.cisco.com/pcgi-bin/cpn/cpn_match_result.pl?perPage=40&CurPosition;=0&Direction;=&ResultType;=EC&search;_id=873856&tab;_name=findsp&SearchType;=Advance&sortBy;=DEFAULT)
http://cisco.com/go/dcuc => Data Center Unified Computing - Partner Central - Cisco Systems(http://www.cisco.com/web/partners/pr11/atp/dcuc/index.html)
http://cisco.com/go/road => Roadways - Transportation - Cisco Systems(http://www.cisco.com/en/US/strategy/transportation/roadways.html)
http://cisco.com/go/used => Cisco Hardware Inspection and Software Re-Licensing Program [Products & Services] - Cisco Systems (http://www.cisco.com/en/US/prod/hw_sw_relicensing_program.html#\~using)
http://cisco.com/go/grid => The Page You Have Requested Is Not Available (http://www.cisco.com/en/US/netsol/ns500/networking_solutions_package.html)
http://cisco.com/go/gold => Cisco.com Login Page (http://www.cisco.com/en/US/partner/partners/pr11/pr8/pr27/partners_pgm_concept_home.shtml)
http://cisco.com/go/ipnd => Cisco.com Login Page (http://www.cisco.com/en/US/partner/partners/pr11/incentive/defender.shtml)
http://cisco.com/go/srnd => Design Zone - Main Page - Cisco - Cisco Systems (http://www.cisco.com/en/US/netsol/ns742/networking_solutions_program_category_home.html)
http://cisco.com/go/isrd => The Page You Have Requested Is Not Available (http://www.cisco.com/web/about/security/security_services/ciag/research/index.html)
http://cisco.com/go/apae => Cisco Application Performance Assurance Engine - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps9799/index.html)
http://cisco.com/go/cuae => Cisco Unified Application Environment - Cisco Systems (http://www.cisco.com/en/US/netsol/ns738/networking_solutions_package.html)
http://cisco.com/go/cmbe => Cisco Unified Communications Manager Business Edition - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps7273/index.html)
http://cisco.com/go/cube => Cisco Unified Border Element (CUBE) - Cisco Systems (http://www.cisco.com/en/US/products/sw/voicesw/ps5640/index.html)
http://cisco.com/go/pace => Compliance - Cisco Systems (http://www.cisco.com/en/US/netsol/ns661/index.html)
http://cisco.com/go/ccde => 502 Proxy Error (http://www.cisco.com/web/learning/le3/ccde/index.html)
http://cisco.com/go/edge => Edge Networks - Cisco Systems (http://www.cisco.com/en/US/netsol/ns592/networking_solutions_solution.html)
http://cisco.com/go/ccie => Cisco Certified Internetwork Expert - CCIE - Cisco Systems(http://www.cisco.com/web/learning/le3/ccie/index.html)
http://cisco.com/go/ccme => Cisco Unified Communications Manager Express(CME) - Cisco Systems (http://www.cisco.com/en/US/products/sw/voicesw/ps4625/index.html)
http://cisco.com/go/hire => IT Managers - The Cisco Learning Network (https://cisco.hosted.jivesoftware.com/community/promo-014-hire?utm_source=tm&utm;_medium=pm&utm;_campaign=promo-014)
http://cisco.com/go/core => Core Networks - Cisco Systems (http://www.cisco.com/en/US/netsol/ns573/networking_solutions_solution.html)
http://cisco.com/go/ahse => Cisco Application-Oriented Networking Healthcare Services Extensions - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps9715/index.html)
http://cisco.com/go/ctwe => Cisco TelePresence WebEx Engage [TelePresence] - Cisco Systems (http://www.cisco.com/en/US/solutions/ns669/webex_engage.html)
http://cisco.com/go/skye => Small Business Empowered by Cisco - Cisco Systems (http://www.cisco.com/web/solutions/smb/heroes/index.html?Referring_site=PrintTv&Country;_Site=us&Campaign;=SMB+Heroes&Position;=Vanity&Creative;=go/skye&Where;=go/skye)
http://cisco.com/go/dcof => The Data Center of the Future - Cisco Systems (http://www.cisco.com/en/US/solutions/ns708/sol_generic_dc_of_the_future.html)
http://cisco.com/go/svig => Introduction - Silicon Valley Impact Grants - Cisco Systems(http://www.cisco.com/web/about/ac48/sv_grants.html)
http://cisco.com/go/fclg => Search Seminars and Webcasts - Events, Webcasts and Seminars - Cisco Systems(http://www.cisco.com/pcgi-bin/sreg2/register/regdetail_private.pl?LANGUAGE=E&METHOD;=E&TOPIC;_CODE=9947&PRIORITY;_CODE=176571_4)
http://cisco.com/go/blog => Architectures and Solutions (http://blogs.cisco.com/ciscotalk/solutions)
http://cisco.com/go/ibsg => Welcome to Cisco IBSG - Internet Business Solutions Group - Cisco Systems(http://www.cisco.com/web/about/ac79/index.html)
http://cisco.com/go/gdsg => Defense - Government - Cisco Systems(http://www.cisco.com/en/US/strategy/government/defense.html)
http://cisco.com/go/ggsg => Defense - Government - Cisco Systems(http://www.cisco.com/web/strategy/government/defense.html)
http://cisco.com/go/nmtg => Network Management - Main Page - Cisco Systems (http://www.cisco.com/en/US/products/sw/netmgtsw/index.html)
http://cisco.com/go/tech => Cisco IOS Technologies - Cisco Systems (http://www.cisco.com/en/US/products/ps6537/products_ios_sub_category_home.html)
http://cisco.com/go/dcni => Advanced Data Center Networking Infrastructure - Partner Central - Cisco Systems(http://www.cisco.com/web/partners/program/specializations/datacenter/dcni/index.html)
http://cisco.com/go/bank => This Content Has Moved - Cisco Systems (http://www.cisco.com/now/bank)
http://cisco.com/go/deal => Cisco.com Login Page (http://www.cisco.com/cgi-bin/front.x/AppTool/controller.cgi)
http://cisco.com/go/rail => Public Transportation - Transportation - Cisco Systems(http://www.cisco.com/en/US/strategy/transportation/rail.html)
http://cisco.com/go/cell => Text Messaging at Cisco - About Cisco - Cisco Systems(http://www.cisco.com/web/about/facts_info/sms_reg_info.html)
http://cisco.com/go/sell => Cisco Systems(http://www.cisco.com/web/partners/sell/index.html)
http://cisco.com/go/cuwl => Cisco Unified Workspace Licensing - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps9156/index.html)
http://cisco.com/go/apam => Cisco Application Performance Assurance Network Module - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps9559/index.html)
http://cisco.com/go/asdm => Cisco Adaptive Security Device Manager - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps6121/index.html)
http://cisco.com/go/cvdm => CiscoWorks CiscoView - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/sw/cscowork/ps4565/index.html)
http://cisco.com/go/cwdm => Cisco CWDM Transceiver Modules - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps6575/index.html)
http://cisco.com/go/dcnm => Cisco Data Center Network Manager - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps9369/index.html)
http://cisco.com/go/tpnm => The Page You Have Requested Is Not Available (http://www.cisco.com/en/US/products/ps9800/Products_Sub_Category_Home.html)
http://cisco.com/go/cuom => Cisco Unified Operations Manager - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps6535/index.html)
http://cisco.com/go/cepm => Cisco Policy Management - Cisco Systems (http://www.cisco.com/en/US/products/ps9519/Products_Sub_Category_Home.html)
http://cisco.com/go/cupm => Provisioning - Cisco - Cisco Systems (http://www.cisco.com/en/US/products/ps7125/index.html)
http://cisco.com/go/wism => Cisco Catalyst 6500 Series/7600 Series Wireless Services Module (WiSM) - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps6526/index.html)
http://cisco.com/go/cusm => Cisco Unified Service Monitor - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps6536/index.html)
http://cisco.com/go/fwsm => Cisco Catalyst 6500 Series Firewall Services Module - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/hw/modules/ps2706/ps4452/index.html)
http://cisco.com/go/mwtm => Cisco Mobile Wireless Transport Manager - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps6472/)
http://cisco.com/go/aibn => The Page You Have Requested Is Not Available (http://www.cisco.com/en/US/partners/pr67/pr29/aibn/solution_home.html)
http://cisco.com/go/edcn => Cisco Support Community: Cisco Support Community (http://forums.cisco.com/eforum/servlet/NetProf;jsessionid=k6x5lkj7q1.SJ2B?page=netprof&CommCmd;=MB%3Fcmd%3Ddisplay_messages%26mode%3Dnew%26location%3D.ee71a00)
http://cisco.com/go/ibpn => Cisco Systems, Inc (http://www.cisco.com)
http://cisco.com/go/ispn => Cisco Systems(http://www.cisco.com/web/partners/sell/industry/index.html)
http://cisco.com/go/evpn => Security Solutions for Enterprise - Cisco Systems (http://www.cisco.com/en/US/netsol/ns340/ns394/ns171/networking_solutions_packages_list.html)
http://cisco.com/go/mvpn => Multicast VPN - Cisco Systems (http://www.cisco.com/en/US/products/ps6651/products_ios_protocol_option_home.html)
http://cisco.com/go/dcsn => Advanced Data Center Storage Networking - Partner Central - Cisco Systems(http://www.cisco.com/web/partners/program/specializations/datacenter/dcsn/index.html)
http://cisco.com/go/ggsn => Cisco Gateway GPRS Support Node - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/sw/wirelssw/ps873/index.html)
http://cisco.com/go/goco => Cisco Systems(http://www.cisco.com/web/partners/sell/technology/collaboration/gocollaborate.html)
http://cisco.com/go/logo => Cisco Brand Center - Doing Business With Cisco - Cisco Systems(http://www.cisco.com/en/US/about/ac50/ac47/about_cisco_brand_center.html)
http://cisco.com/go/demo => Cisco Systems(http://www.cisco.com/web/partners/sell/technology/ipc/integrated-solutions/dmr.html)
http://cisco.com/go/cspo => Security Programs - Security Center - Cisco Systems(http://www.cisco.com/web/about/security/cspo/index.html)
http://cisco.com/go/pvso => We Apologize - 401 Error(http://www.cisco.com/partner/services/pvso/)
http://cisco.com/go/dcap => Data Center Assurance Program - Cisco Systems (http://www.cisco.com/en/US/netsol/ns758/networking_solutions_sub_program_home.html)
http://cisco.com/go/asap => The Page You Have Requested Is Not Available (http://www.cisco.com/web/partners/pr192/sp_asap.html)
http://cisco.com/go/csbp => The Page You Have Requested Is Not Available (http://www.cisco.com/en/US/netsol/ns671/networking_solutions_package.html)
http://cisco.com/go/dhcp => Dynamic Host Control Protocol (DHCP)/Domain Name System (DNS) - Cisco Systems (http://www.cisco.com/en/US/products/ps6641/products_ios_protocol_option_home.html)
http://cisco.com/go/mscp => Cisco Systems(http://www.cisco.com/web/partners/pr11/mscp/index.html)
http://cisco.com/go/oscp => Outsourcing Channel Program - Partner Central - Cisco Systems(http://www.cisco.com/web/partners/pr11/outsourcing/index.html)
http://cisco.com/go/mldp => IP Multicast - Cisco Systems (http://www.cisco.com/en/US/products/ps6552/products_ios_technology_home.html)
http://cisco.com/go/ctdp => Introduction - Cisco Technology Developer Program - Cisco Systems(http://www.cisco.com/en/US/partners/pr46/tdp/index.shtml)
http://cisco.com/go/bbip => CCO Decommission Page (http://www.cisco.com/warp/public/732/bbip/)
http://cisco.com/go/clip => Cisco Systems(http://www.cisco.com/web/ordering/ciscocapital/o45/ordering_finance_solution_program0900aecd800ddd5f.html)
http://cisco.com/go/grip => High Availability - Cisco Systems (http://www.cisco.com/en/US/products/ps6550/products_ios_technology_home.html)
http://cisco.com/go/ctmp => Cisco Systems(http://www.cisco.com/web/partners/pr11/incentive/tmp/index.html)
http://cisco.com/go/coop => Cisco Coop Fund Builder (http://www.coams.com/ciscocoop)
http://cisco.com/go/mtop => Radio Access Networks - Cisco Systems (http://www.cisco.com/en/US/netsol/ns675/networking_solutions_solution_category.html)
http://cisco.com/go/clsp => Cisco Learning Solutions Partner - Learning Partners Program Overview - Cisco Systems(http://www.cisco.com/web/learning/le27/le53/learning_partner_clsp.html)
http://cisco.com/go/cusp => Cisco Unified SIP Proxy - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps10140/index.html)
http://cisco.com/go/cptp => Cisco Partner Talent Network (https://secure.partnertalentportal.com/emerging/admin9021/login.asp)
http://cisco.com/go/ccvp => CCVP - Career Certifications & Paths - Cisco Systems(http://www.cisco.com/en/US/learning/le3/le2/le37/le65/learning_certification_type_home.html)
http://cisco.com/go/rsvp => Cisco RSVP Agent - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps6832/index.html)
http://cisco.com/go/nbar => Network Based Application Recognition (NBAR) - Cisco Systems (http://www.cisco.com/en/US/products/ps6616/products_ios_protocol_group_home.html)
http://cisco.com/go/eccr => Experience Cisco Collaboration Roadshow - Cisco Unified Communications & WebEx - Cisco Systems(http://www.cisco.com/web/partners/sell/technology/ipc/announcements/uc7roadshow.html)
http://cisco.com/go/dcdr => Cisco Systems(http://www.cisco.com/web/partners/pr11/incentive/euro/dcdr.html)
http://cisco.com/go/iaas => Infrastructure as a Service - Cisco Systems (http://www.cisco.com/en/US/netsol/ns995/networking_solutions_solution_category.html)
http://cisco.com/go/waas => WAN Optimization - Cisco Systems (http://www.cisco.com/en/US/products/ps5680/Products_Sub_Category_Home.html)
http://cisco.com/go/dcas => Data Center Application Services - Cisco Systems (http://www.cisco.com/en/US/products/ps5719/Products_Sub_Category_Home.html)
http://cisco.com/go/cabs => Cisco.com Login Page (http://forums.cisco.com/eforum/servlet/CAB?page=main&sn;=CAB)
http://cisco.com/go/mibs => Cisco IOS MIB Locator (http://tools.cisco.com/ITDIT/MIBS/servlet/index)
http://cisco.com/go/isbs => Design Zone for Branch - Cisco Systems (http://www.cisco.com/en/US/netsol/ns816/networking_solutions_program_home.html)
http://cisco.com/go/sbcs => Cisco Smart Business Communications System - Cisco Systems(http://www.cisco.com/cisco/web/solutions/small_business/products/voice_conferencing/smart_business_communications_system/index.html?Referring_site=PrintTv&Country;_Site=us&Campaign;=SAMBA&Position;=Vanity&Creative;=go/sbcs&Where;=go/sbcs)
http://cisco.com/go/hucs => Cisco Hosted Unified Communications Services - Cisco Systems (http://www.cisco.com/en/US/netsol/ns757/networking_solutions_solution.html)
http://cisco.com/go/wids => Cisco Adaptive Wireless IPS Software - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps9817/index.html)
http://cisco.com/go/bugs => Cisco.com Login Page (http://tools.cisco.com/Support/BugToolKit/action.do?hdnAction=searchBugs)
http://cisco.com/go/iris => Cisco Internet Routing in Space (IRIS) - Industry Solutions - Cisco Systems(http://www.cisco.com/web/strategy/government/space-routing.html)
http://cisco.com/go/mpls => Multiprotocol Label Switching (MPLS) - Cisco Systems (http://www.cisco.com/en/US/products/ps6557/products_ios_technology_home.html)
http://cisco.com/go/vams => Cisco Video Assurance Management Solution - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps9518/index.html)
http://cisco.com/go/cpms => Welcome - Cisco Systems(http://www.cisco.com/web/learning/le21/le34/cpnmarketing/2007/)
http://cisco.com/go/lpms => Cisco.com Login Page (http://tools.cisco.com/E-Learning-IT/LPCM/jsp/LpcmWelcome.jsp)
http://cisco.com/go/ibns => Identity Based Networking Services - Cisco Systems (http://www.cisco.com/en/US/products/ps6638/products_ios_protocol_group_home.html)
http://cisco.com/go/nxos => Cisco NX-OS Software - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps9372/index.html)
http://cisco.com/go/nips => News@Cisco -> Executive Biographies (http://newsroom.cisco.com/dlls/tln/research_studies/nips/index.html)
http://cisco.com/go/wips => Cisco Adaptive Wireless IPS Software - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps9817/index.html)
http://cisco.com/go/apps => Cisco Systems: Unified Communications Applications Central (http://forums.cisco.com/eforum/servlet/IPCApps?page=main)
http://cisco.com/go/mars => Cisco Security Monitoring, Analysis, and Response System (MARS) - Cisco Systems - Cisco Systems (http://www.cisco.com/en/US/products/ps6241/index.html)
http://cisco.com/go/nbss => The Page You Have Requested Is Not Available (http://www.cisco.com/en/US/netsol/ns614/networking_solutions_sub_solution.html)
http://cisco.com/go/ucss => Cisco Unified Communications Software Subscription (UCSS) - Cisco Systems (http://www.cisco.com/en/US/products/ps9158/index.html)
http://cisco.com/go/gdss => 502 Proxy Error (http://www.cisco.com/en/US/strategy/government/defense.html)
http://cisco.com/go/ciss => Cisco Feature Navigator - Cisco Systems (http://tools.cisco.com/ITDIT/ISTMAIN/jsp/index.jsp)
http://cisco.com/go/cnss => 07/01/03 - Recent Program Information - Cisco Systems(http://www.cisco.com/web/learning/le3/whats_new/infosec.html)
http://cisco.com/go/skus => Cisco Systems(http://www.cisco.com/web/partners/pr11/incentive/eligible_skus.html)
http://cisco.com/go/news => News@Cisco -> News@Cisco (http://newsroom.cisco.com/dlls/index.html)
http://cisco.com/go/csat => Customer Satisfaction - Cisco Systems(http://www.cisco.com/web/partners/pr11/pr20/partners_customer_satisfaction_concept_home.html)
http://cisco.com/go/cebt => Collaboration Enabled Business Transformation - Cisco Systems (http://www.cisco.com/en/US/netsol/ns952/index.html)
http://cisco.com/go/msft => DC Partner - Microsoft - Cisco Systems (http://www.cisco.com/en/US/netsol/ns963/index.html)
http://cisco.com/go/lcmt => Cisco.com Login Page (http://tools.cisco.com/GET/lrncrd/jsp/index.jsp)
http://cisco.com/go/cart => Log-In (http://resources.cisco.com/app/tree.taf?asset_id=175513&public;_view=true)
http://cisco.com/go/srst => Cisco Unified Survivable Remote Site Telephony - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/sw/voicesw/ps2169/index.html)
http://cisco.com/go/issu => In-Service Software Upgrade (ISSU) - Cisco Systems (http://www.cisco.com/en/US/products/ps7149/products_ios_protocol_group_home.html)
http://cisco.com/go/srsv => Cisco Survivable Remote Site Voicemail - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps10769/index.html)
http://cisco.com/go/iptv => IPTV Solutions - Cisco Systems (http://www.cisco.com/en/US/netsol/ns610/networking_solutions_solution_category.html)
http://cisco.com/go/rfgw => Cisco RF Gateway Series - Products & Services - Cisco Systems (https://www.cisco.com/en/US/products/ps8360/index.html)
http://cisco.com/go/grow => The Page You Have Requested Is Not Available (http://www.cisco.com/web/partners/grow.html)
http://cisco.com/go/cbsw => Cisco Systems(http://www.cisco.com/web/partners/sell/smb/university/training.html)
http://cisco.com/go/uccx => Cisco Unified Contact Center Express - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/sw/custcosw/ps1846/index.html)
http://cisco.com/go/amex => The Page You Have Requested Is Not Available (http://www.cisco.com/en/US/netsol/ns339/networking_solutions_small_medium_sized_business_home.html)
http://cisco.com/go/gray => Cisco Hardware Inspection and Software Re-Licensing Program [Products & Services] - Cisco Systems (http://www.cisco.com/en/US/prod/hw_sw_relicensing_program.html#\~using)
http://cisco.com/go/grey => Cisco Hardware Inspection and Software Re-Licensing Program [Products & Services] - Cisco Systems (http://www.cisco.com/en/US/prod/hw_sw_relicensing_program.html#\~using)
http://cisco.com/go/easy => Cisco Embedded Automation Systems - Cisco Systems (http://www.cisco.com/en/US/products/ps10777/products_ios_protocol_group_home.html)
http://cisco.com/go/4200 => Cisco IPS 4200 Series Sensors - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/hw/vpndevc/ps4077/index.html)
http://cisco.com/go/4500 => Cisco Catalyst 4500 Series Switches - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/hw/switches/ps4324/index.html)
http://cisco.com/go/6500 => Cisco Catalyst 6500 Series Switches - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/hw/switches/ps708/index.html)
http://cisco.com/go/3700 => Cisco -Cisco 3700 Series Routers (http://www.cisco.com/warp/public/cc/pd/rt/ps282/)
http://cisco.com/go/3800 => Cisco 3800 Series Integrated Services Routers - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps5855/index.html)
http://cisco.com/go/1800 => Cisco 1800 Series Integrated Services Routers - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps5853/index.html)
http://cisco.com/go/2800 => Cisco 2800 Series Integrated Services Routers - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps5854/index.html)
http://cisco.com/go/4900 => Cisco Catalyst 4900 Series Switches - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps6021/index.html)
http://cisco.com/go/vc10 => VoiceCon 2010 - Cisco Events - Cisco Systems(http://www.cisco.com/web/learning/le21/le34/voicecon/2010/index.html)
http://cisco.com/go/tv10 => Telepresence - Overview - Cisco Systems (http://www.cisco.com/en/US/netsol/ns669/networking_solutions_solution_segment_home.html)
http://cisco.com/go/1520 => Cisco Aironet 1520 Series - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps8368/index.html)
http://cisco.com/go/3750 => Cisco Catalyst 3750 Series Switches - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/hw/switches/ps5023/index.html)
http://cisco.com/go/2950 => Cisco Catalyst 2950 Series Switches - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/hw/switches/ps628/index.html)
http://cisco.com/go/3560 => Cisco Catalyst 3560 Series Switches - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/hw/switches/ps5528/index.html)
http://cisco.com/go/2960 => Cisco Catalyst 2960 Series Switches - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps6406/index.html)
http://cisco.com/go/cpi1 => Cisco.com Login Page (http://www.cisco.com/en/US/partner/partners/pr47/newsletter/us.shtml)
http://cisco.com/go/cio1 => Preparing Business for the Upturn with IT [Products & Services] - Cisco Systems (http://www.cisco.com/en/US/solutions/ns340/ns856/ns870/prep_business_for_upturn_generic.html?POSITION=PrintVanity&COUNTRY;_SITE=us&CAMPAIGN;=EMME—CIO+Awarenett&CREATIVE;=go/cio1)
http://cisco.com/go/tv11 => Telepresence - Overview - Cisco Systems (http://www.cisco.com/en/US/netsol/ns669/networking_solutions_solution_segment_home.html)
http://cisco.com/go/7921 => Cisco Unified Wireless IP Phone 7921G - Cisco Systems (http://www.cisco.com/en/US/products/ps7071/index.html)
http://cisco.com/go/7931 => Cisco Unified IP Phone 7931G - Cisco Systems (http://www.cisco.com/en/US/products/ps7062/index.html)
http://cisco.com/go/1861 => Cisco 1861 Integrated Services Router - Cisco Systems (http://www.cisco.com/en/US/products/ps8321/index.html)
http://cisco.com/go/cpi2 => Cisco.com Login Page (http://www.cisco.com/en/US/partner/partners/pr47/newsletter/us.shtml)
http://cisco.com/go/cio2 => Preparing Business for the Upturn with IT [Products & Services] - Cisco Systems (http://www.cisco.com/en/US/solutions/ns340/ns856/ns870/prep_business_for_upturn_generic.html?POSITION=PrintVanity&COUNTRY;_SITE=us&CAMPAIGN;=EMME%2D%2DCIO+Awarenett&CREATIVE;=go/cio2)
http://cisco.com/go/tv12 => Telepresence - Overview - Cisco Systems (http://www.cisco.com/en/US/netsol/ns669/networking_solutions_solution_segment_home.html)
http://cisco.com/go/cpi3 => Cisco.com Login Page (http://www.cisco.com/en/US/partner/partners/pr47/newsletter/us.shtml)
http://cisco.com/go/cio3 => Preparing Business for the Upturn with IT [Products & Services] - Cisco Systems (http://www.cisco.com/en/US/solutions/ns340/ns856/ns870/prep_business_for_upturn_generic.html?POSITION=PrintVanity&COUNTRY;_SITE=us&CAMPAIGN;=EMME%2D%2DCIO+Awarenett&CREATIVE;=go/cio3)
http://cisco.com/go/tv13 => Telepresence - Overview - Cisco Systems (http://www.cisco.com/en/US/netsol/ns669/networking_solutions_solution_segment_home.html)
http://cisco.com/go/cpi4 => Cisco.com Login Page (http://www.cisco.com/en/US/partner/partners/pr47/newsletter/us.shtml)
http://cisco.com/go/cio4 => Preparing Business for the Upturn with IT [Products & Services] - Cisco Systems (http://www.cisco.com/en/US/solutions/ns340/ns856/ns870/prep_business_for_upturn_generic.html?POSITION=PrintVanity&COUNTRY;_SITE=us&CAMPAIGN;=EMME%2D%2DCIO+Awarenett&CREATIVE;=go/cio4)
http://cisco.com/go/tv14 => Telepresence - Overview - Cisco Systems (http://www.cisco.com/en/US/netsol/ns669/networking_solutions_solution_segment_home.html)
http://cisco.com/go/9124 => Cisco MDS 9124 Multilayer Fabric Switch - Cisco Systems (http://www.cisco.com/en/US/products/ps7079/index.html)
http://cisco.com/go/cpi5 => Cisco.com Login Page (http://www.cisco.com/en/US/partner/partners/pr47/newsletter/us.shtml)
http://cisco.com/go/cio5 => Preparing Business for the Upturn with IT [Products & Services] - Cisco Systems (http://www.cisco.com/en/US/solutions/ns340/ns856/ns870/prep_business_for_upturn_generic.html?POSITION=PrintVanity&COUNTRY;_SITE=us&CAMPAIGN;=EMME%2D%2DCIO+Awarenett&CREATIVE;=go/cio5)
http://cisco.com/go/nw05 => Cisco Systems - Redirect to Networkers (http://www.cisco.com/offer/nwol04/128101_1)
http://cisco.com/go/tv15 => Telepresence - Overview - Cisco Systems (http://www.cisco.com/en/US/netsol/ns669/networking_solutions_solution_segment_home.html)
http://cisco.com/go/2955 => Cisco Catalyst 2955 Series Switches - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps6738/index.html)
http://cisco.com/go/2975 => Cisco Catalyst 2975 Series Switches - Products & Services - Cisco Systems (http://www.cisco.com/en/US/products/ps10081/index.html)
http://cisco.com/go/cpi6 => Cisco.com Login Page (http://www.cisco.com/en/US/partner/partners/pr47/newsletter/us.shtml)
http://cisco.com/go/cio6 => Preparing Business for the Upturn with IT [Products & Services] - Cisco Systems (http://www.cisco.com/en/US/solutions/ns340/ns856/ns870/prep_business_for_upturn_generic.html?POSITION=PrintVanity&COUNTRY;_SITE=us&CAMPAIGN;=EMME%2D%2DCIO+Awarenett&CREATIVE;=go/cio6)
http://cisco.com/go/cgv6 => Cisco Carrier-Grade IPv6 Solution - Cisco Systems (http://www.cisco.com/en/US/netsol/ns1017/networking_solutions_solution_category.html)
http://cisco.com/go/nw06 => Cisco Systems - Redirect to Networkers (http://www.cisco.com/offer/nwcdcpktads/133543_4)
http://cisco.com/go/tv16 => Telepresence - Overview - Cisco Systems (http://www.cisco.com/en/US/netsol/ns669/networking_solutions_solution_segment_home.html)
http://cisco.com/go/cpi7 => Cisco.com Login Page (http://www.cisco.com/en/US/partner/partners/pr47/newsletter/us.shtml)
http://cisco.com/go/cio7 => Preparing Business for the Upturn with IT [Products & Services] - Cisco Systems (http://www.cisco.com/en/US/solutions/ns340/ns856/ns870/prep_business_for_upturn_generic.html?POSITION=PrintVanity&COUNTRY;_SITE=us&CAMPAIGN;=EMME%2D%2DCIO+Awarenett&CREATIVE;=go/cio7)
http://cisco.com/go/tv17 => Telepresence - Overview - Cisco Systems (http://www.cisco.com/en/US/netsol/ns669/networking_solutions_solution_segment_home.html)
http://cisco.com/go/cpi8 => Cisco.com Login Page (http://www.cisco.com/en/US/partner/partners/pr47/newsletter/us.shtml)

Parsing pcap files with Perl

Attila-Mihaly Balazs — Fri, 19 Mar 2010 15:43:00 +0200

Recently I was reading the blogpost on the BrekingPoint labs log about parsing pcap files with Perl and I immediately said to myself: it is impossible that there isn’t a module on CPAN, because Perl is great. Turns out I was right, there is Net::TcpDumpLog which can be combined with the NetPacket family of modules to parse the higher level protocols. Because example code is rather sparse on the POD pages of the respective modules, here is a small example to illustrate their use:

use Net::TcpDumpLog;
use NetPacket::Ethernet;
use NetPacket::IP;
use NetPacket::TCP;
use strict;
use warnings;

my $log = Net::TcpDumpLog->new(); 
$log->read("foo.pcap");

foreach my $index ($log->indexes) { 
  my ($length_orig, $length_incl, $drops, $secs, $msecs) = $log->header($index); 
  my $data = $log->data($index);

  my $eth_obj = NetPacket::Ethernet->decode($data);    
  next unless $eth_obj->{type} == NetPacket::Ethernet::ETH_TYPE_IP;

  my $ip_obj = NetPacket::IP->decode($eth_obj->{data});
  next unless $ip_obj->{proto} == NetPacket::IP::IP_PROTO_TCP;

  my $tcp_obj = NetPacket::TCP->decode($ip_obj->{data});
  my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime($secs + $msecs/1000);
  print sprintf("%02d-%02d %02d:%02d:%02d.%d", 
    $mon, $mday, $hour, $min, $sec, $msecs), 
    " ", $eth_obj->{src_mac}, " -> ", 
    $eth_obj->{dest_mac}, "\n";    
  print "\t", $ip_obj->{src_ip}, ":", $tcp_obj->{src_port}, 
    " -> ", 
    $ip_obj->{dest_ip}, ":", $tcp_obj->{dest_port}, "\n";
}

The code does the following: it opens the pcap file named “foo.pcap”, iterates over all the packets (it assumes that they all are Ethernet packets) and looks for TCP packets. Finally it prints out some information about these packets (capture time, source/destination mac, source/destination ip:port). You can customize it to fit your needs.

Small, somewhat offtopic rant: one should always think at least twice before publishing code which does such elementary things. Find a library and use it. If it doesn’t work, try patching it so that it works and send back the code to the original author. Only if this fails should you start from scratch.

Reusing existing code has many advantages: from your point of view, you can be sure that you can get code which worked for a couple of people. This is especially true for Perl modules which have a strong culture of testing. Also, even these “simple” problems like parsing a TCP packet have many corner cases which you will almost certainly miss at the first go, and as a result, half of your time will be spent hunting them down and only half of your time will be dedicated to solving the actual problem (this is if you are lucky – if you are unlucky, your code will skip over the special cases and it may make your entire analysis irrelevant).

Looking at it from the other side we have: more concentration of the way to do “X” means that the code will be more tested, leading it to be used more, meaning that it will be better tested and thus creating a positive feedback loop. Also, if you believe in the open-source ethos (and supposedly you do, since you published your code in the first place), you should consider maximizing the return while minimizing the effort needed.

Picture taken from greyloch’s photostream with permission.

Update: updated NetPacket link - thank you Anonymous.

Spammy Mike

Attila-Mihaly Balazs — Fri, 19 Mar 2010 15:08:00 +0200

While most of the time I simply skip / delete any malicious content encountered, from time to time I do some quick investigation on items which peak my interest. For example the following comment was posted on a friends blog:

You make a good point, and it is one I often make about encryption. There are just too many standards out there for any smooth communication to occur. I think there are some companies who are getting it right with their approach to malware, but many malware just can’t seem to get their fundamentals down.

I didn’t remove the links, since they point to complete benign sites (sophos.com and kaspersky.com). Mike’s profile is private, but a quick search shows many other spammy comments. Unfortunately there doesn’t seem to be a way to report individual Blogger users as spammers, just actual blogs.

BTW. the same comment spam seems to have hit at least one other security blog. From the screenshot it seems that the spammer also uses the Blogger name MikeFrizzi, which seems to be linked to a real person, but then again, it is quite easy to create realistically looking “shadow identities” for people by scraping other websites.

This is as much as a quick search revealed and I would like to leave you with the following thoughts:

Do comment moderation, at least retroactively if not proactively (small plug: I do moderate comments, but for the ones I approve the username links are without the nofollow tag – as per the u comment, i follow “ethos”)
There is very little certainty on the Internet. Just because someone claims to be somebody (like the MikeFrizzi profile), it doesn’t mean he actually is that person.
Also, the link between spam and the actual company being promoted is hard to prove. I don’t think that Sophos or Kaspersky were spamming here directly, but I do think it’s possible that some remotely connected company (ie. something along the lines of “a company hired by the outsourced marketing department”) did in fact employ such dubious (and useless, since in Blogger all the links in comments are “nofollow’ed”) techniques.
Or, it may be, that some blackhats want to give the impression that these companies are spamming to erode their credibility…

Update: Sophos confirmed that it was a run-amok “marketing” company hired by them who posted the spam.

Picture taken from madmarv00’s photostream with permission.

Fixing a dead Asus WL-500g

Attila-Mihaly Balazs — Wed, 17 Mar 2010 10:16:00 +0200

A short story with a happy ending: my Asus WL-500g locked up and it wasn’t starting, even after I hard-reset it (removed the power and plugged it back in). All the LEDs were constantly on (normally, when you plug in the power, they should light up for a second or so and the turn off). After some searching I found a page on FixYa which pointed me to the following forum thread: Dead or brick ? Lan and WAN 1-4 leds on steady. There were a couple of stories here with identical symptoms, so I decided to give the solution a try.

The tricky part was to find a source which could deliver 2.5A at 5V (most of the ones I found peaked at 1.5A). Finally I found this one locally and it worked like a charm! (It seems the trick is to search for “switch power source”). Also, for extra safety, look carefully at the adapter when you get it to make sure that it conforms to the specification (5V DC / 2.5A) and if possible, use a multimeter to ensure that the polarization is correct (in my case it was + on the inside, – on the outside).

PS. I didn’t get to disassemble my old source yet, but I will post photos whenever I do so.

Computing the last digit of b^e efficiently

Attila-Mihaly Balazs — Mon, 15 Mar 2010 13:20:00 +0200

Geek PSA: Yesterday was PI day (3.14, get it?). Lets celebrate with this spiked math comic:

Last week I saw the following problem which peaked my interest:

Compute the last (decimal) digit of 2 raised to the power e where e might be very large.

We assume that we are talking about positive, integer exponents here. The key observation here is that the last digit form a cycle:

...2 * 2 = ...4
...4 * 2 = ...8
...8 * 2 = ...6
...6 * 2 = ...2

So you can compute the last digit by calculating e MOD 4, which gives us the position in the cycle. Next, I wondered if all the digits have this cycling going on, so I searched around a little bit and found this page which gives the cycle for all ten digits. Now you can compute the last digit of the formula b\^e with the following algorithm:

Take the last digit of b
compute e MOD (length of cycle for last digit of b)
return the given element of the cycle

This is very nice, since we can operate on arbitrary sized b, but we still need to be able to perform the modulo operation on e, which might be inconvenient if e is large. Fortunately we can make the following observation:

The length of the cycle is either 1, 2 or 4
For x in [1,2,4] you have …ab MOD x = ab MOD x (ie you can take only the last two digits and compute the modulo) since 100 is a multiple of both 2 and 4 (meaning that a number in the form of …00 will always be divisible by both 2 and 4, hence it contributes nothing to the modulo operation)

So here is the final algorithm which works quickly even for very large values of b and e

Take the last digit of b
compute e MOD length_of_cycle by using the last two digits of e
return the given element of the cycle

Give it a try below if you have javascript enabled:

Math is fun :-).

Finally, I would like to leave you with the following question: is it possible to efficiently compute an arbitrary digit of b\^e? My intuition is that there are cycles in all of them, however they might be quite long…

In praise of Regexp::Assemble

Attila-Mihaly Balazs — Mon, 15 Mar 2010 10:42:00 +0200

…and of the Perl modules in general. I had the following problem:

Given a list of 16 character alphanumeric IDs, find all the lines from a large-ish (\~6GB) logfile which contain at least one of the IDs.

The naive approach was to construct a big regular expression like \W(\QID1\E|\QID2\E|\QID3\E...)\W and match it against every line (I needed to capture the actual ID to know in which bucket to place the line). Needless to say, as it is the case with most naive approaches, it was slooooow (basically, it was hogging the CPU, not the disk). So, by searching around a little bit I found Regexp::Optimizer and Regexp::Assemble. Of the two the later seemed the more mature one, so – after quickly installing it with CPAN – I’ve put it into my code and made it run at the “speed of the disk”. W00t! Perl + CPAN + clever modules rock!

PS. A little benchmark data (take it with a grain of salt, since you should be profiling not benchmarking most of the time):

Unoptimized regex size: 873 427 characters
Optimized regex: 69 536 characters
Unoptimized regex matchtime over 380 MB of data: \~1.9 hours (which would mean a throughput of \~58KB / sec – well below disk speed)
Optimized regex matching over the same 380 MB of data: 2 sec (throughput: 190 MB/sec !!!)

How cool is this?

Funny pictures post :-)

Attila-Mihaly Balazs — Mon, 15 Mar 2010 10:00:00 +0200

About DRM (this is floating around the interwebs, I’m not entirely sure about its origin):

The second one is from a webcomic (added to my reader :-)) which I discovered via Bruce Schneier’s blog:

And finally, for all the programmers out there (found it via The Reinvigorated Programmer’s blog):

RegEx which matches strings not containing a substring

Attila-Mihaly Balazs — Wed, 10 Mar 2010 14:08:00 +0200

This is an interesting problem which can appear in certain cases (although not very often). A little searching around led me to many posts stating that there is no easy solution and the following easy solution:

^((?!my string).)*$

It works as follows: the matching string must contain zero or more characters which are not preceded (?! is the negative look behind operator) by the given string.

It is quite straight-forward, uses operators which are widely supported by regualar expression engines and works even if “my string” is at the end of the string we are trying to match – for reasons which are not entirely clear to me.

Obviously it is a hack and you shouldn’t use it if you can use a clearer way to indicate your intention, but it is a nifty tool to have in your toolbox for that one moment when you need it.

Picture taken from crowt59’s photostream with permission.

Java import statement gotcha

Attila-Mihaly Balazs — Wed, 10 Mar 2010 13:49:00 +0200

There is a lot of debate on the intertubes if one should or shouldn’t use wildcard imports. I’m mostly indifferent to the discussion (mainly because all the package references are resolved compile time – so there is no performance overhead – and because today’s IDE’s contain a lot of smarts to help you figure out which is the actual class being referenced), but here is something interesting I discovered recently:

Lets say that we have two classes with the same name in different packages:

package foo;

public class Foo {
    public static void print() {
        System.out.println("Foo");
    }
}

package bar;

public class Foo {
    public static void print() {
        System.out.println("Bar");
    }
}

Now create a test class in the bar package:

package bar;

import foo.*;

public class Test {
    public static void main(String[] args) {
        Foo.print();
    }
}

Question: what will this class print out? The answer is – surprisingly for me - “Bar”. It seems that the Java compiler (tested with 1.6u18, but this is probably the same with other versions – although I’m not sure about alternative implementations like GCJ) uses the following order to determine the canonical class name:

Classes which are explicitly imported
Classes which are in the current package
Classes imported with wildcards

Just something to know about.

Picture taken from salimfadhley’s photostream with permission.

PS. Amusing sidenote: when you search for import on Flickr, a lot of “babe” photos come up, even with safe search on. Is “import” a codeword for something? :-)

FindersUK review

Attila-Mihaly Balazs — Thu, 25 Feb 2010 14:40:00 +0200

Finding relatives was always a topic which interested me marginally, so I was interested in checking out the Finders UK website (which BTW, can also be read in other ways because URLs don’t have the concept of capitalization implicitly – lets just leave at that). They offer different services like tracing heirs. Unfortunately (for me) their services are limited to the UK (then again, it makes the company much more compelling, since supposedly a company can offer the best services by specializing in one core competency).

They don’t have a public price offer, but this is understandable since the amount of work involved can vary wildly between cases. All in all this looks like a good company if you need such services in the UK (the links on their webpage to different industry associations are all legitimate, even though some of them a little out of date). They also support two non-profit organizations: the Marie Curie Cancer care and Missing People, which is very generous of them.

Full disclosure: this is a paid review from ReviewMe. Under the terms of the understanding I was not obligated to skew my viewpoint in any way (ie. only post positive facts).

BrickHouse Alert review

Attila-Mihaly Balazs — Thu, 25 Feb 2010 11:00:00 +0200

After looking at the site of this Medical Alert company I’m left with troubling questions in my mind. Having an ill grand-parent makes me acutely appreciate the need for Medical Alert services, but this company leaves many open questions, especially considering the fact that this is a company which requires a monthly subscription. Their site is just a front for Yahoo Stores (which isn’t bad in itself, since it makes business sense to outsource the non-core competencies of your activity – like running a website) and the registration data for the site is protected by “Whois Guard” (again, something which makes sense in a lot of situations, but is just an other stab in the trustworthiness of the company). They also don’t seem to provide any physical address or concrete information about the personal used in their call centers.

All in all, I wouldn’t do business with this company unless somebody I know and trust recommends them. While the issues raised might very well be just an oversight on the part of the company, it suggests a lack of understanding of consumer psychology which is alarming given the sensitive nature of the issue (health care).

Full disclosure: this is a paid review from ReviewMe. Under the terms of the understanding I was not obligated to skew my viewpoint in any way (ie. only post positive facts).

Globe Runner review

Attila-Mihaly Balazs — Tue, 16 Feb 2010 16:49:00 +0200

The second review for today is also for a SEO Consultant: Globe Runner SEO. From all that I can gather, they are a legitimate company (here is their BBB page - so hard to tell these days :-( ), and they also run BabySafe Travel. They also don’t seem to come up in the first 10 pages (!) on Google when I’ve searched for SEO. Then again, this might be a good thing, since these days many of the top spots for any search are dominated by scammers (though not the top one usually).

They also have an informative blog with original content. In the end it everyone must make the decision about needing SEO help by herself / himself. The technology is not very complicated behind most of the actions and a good website should already have most of the elements, but if you not a “webhead”, you will benefit from the advice of an expert. One thing I missed on their site is the clear display of prices for their services, but then again this can be interpreted in a positive manner as: “we won’t go into an one-sided competition on price, because we should be judged on more than one figure” (my words, not theirs :-)).

Full disclosure: this is a paid review from ReviewMe. Under the terms of the understanding I was not obligated to skew my viewpoint in any way (ie. only post positive facts).

Softline Local Review

Attila-Mihaly Balazs — Tue, 16 Feb 2010 16:32:00 +0200

Many say that local online marketing is the “next frontier”. While I have some doubts with regarding this phenomenon (my money is on the ever expanding global economy – even though I’ve got “slapped” a couple of times by companies refusing to deliver to Romania), I’ve looked at softline local.

They seem like a company to consider if you wish to include such tool in your arsenal. Their main target is the USA (understandably so, since they are located in LA), so this probably isn’t for you if you are outside of the USA or your services are not location dependent.

In case you do decide to go with them, I would recommend to check out in detail their pricing and maybe inquire about the details of each element in their price chart. While all of them seem like a good idea (I especially liked the inclusion in GPS systems), I would question the repeated value of some of them (maybe some of these services require an annual fee – but many of them – like the inclusion in search indexes – definitely don’t). In conclusion, my final advice is: the services look promising, but proceed with caution, maybe do a short-term evaluation of their services.

Full disclosure: this is a paid review from ReviewMe. Under the terms of the understanding I was not obligated to skew my viewpoint in any way (ie. only post positive facts).

You go PHD Comics!

Attila-Mihaly Balazs — Tue, 26 Jan 2010 18:39:00 +0200

PHD Comics is always great and hilarious (and worth to subscribe to if you are even vaguely related to the academic world - like trough a friend of a friend :-)) but there are those occasions when it is epic, like this one:

![](http://www.phdcomics.com/comics/archive/phd012010s.gif)

The media can almost never be trusted to get things right and we should get into the habit of questioning everything they deliver. Think for yourself people, get a grip on basic math and logic and learn how to dig up information!

PS. I’m not “new media”, I’m just a raindrop :-p.

Carving out files with Perl

Attila-Mihaly Balazs — Tue, 26 Jan 2010 16:11:00 +0200

I’ve had to use this trick a couple of times the last few years, so I decided that I might as well document it:

If you have an image of a storage media (like an SD card or CD/DVD) which you can not mount (either because the filesystem is hosed - that’s a technical term for damaged beyond repair :-) - or because it uses some proprietary extension - *cough* MS *cough) and you know the approximate size of each file (maybe they are JPEGs or AVIs), you could adapt this script of mine.

What it does:

It reads $search_buffer bytes from $input
It looks for $header (as it is written it looks for RIFF, which means AVI or WAV usually - for JPEG you would use “\xFF\xD8”)
If it finds it, it dumps $extracted_size bytes from the given position (this should be set to be larger than the biggest file you expect)
It not, it seeks forward $search_buffer - length($header) (to handle the cases when the header is split by the border of the buffer)

The script is not perfect (for one it tries to load the entire file into memory before writing it out; it also doesn’t do any validation of the fileformat, thus possibly creating some garbage output), but it worked well for me in the past, so I thought I share it.

PS. If you need some more serious file recovery, you might want to look at PhotoRec and TestDisk. They are both free (as in freedom - GPL license) and seem to be great programs (I never actually managed to get them to recover more than my little cobbled together script, but I might have some very particular usecases).

Should I use RoboForm?

Attila-Mihaly Balazs — Mon, 25 Jan 2010 13:58:00 +0200

A little background about this post: while I was vaguely aware of RoboForm, I never took a closer look at it until I saw the following post by Derek: Beware of fake shopping sites. What it basically says is that a password manager (such as Roboform) will help you avoid phising sites, because it will observe the different URL and will not pre-fill your contact details, thus providing you with an additional warning sign that something is phishy :-)

I have a great deal of respect for Derek so I did some quick checking around the product. Here are my findings:

My first question was: how does the company behind Roboform make money? Because if they don’t have a clear way of making money and they are giving away the product for free, you might start to wonder - aren’t they giving away my information to make money? I was relieved to find that they have a pro version which costs money. This doesn’t necessarily mean that they aren’t giving away your information, but there is one less reasons for them to do so.
The animated banner ad reminded me of some “smiley toolbar” advertisements which lead to adware, but taste is relative :-). An other marketing method of them which I personally found distasteful (although it is certainly legal) is the “Free Password Scan“. This is a small tool which, when executed, shows the saved passwords from IE and Firefox. The purpose of this - I assume - is to show how “insecure” those browsers are (BTW, if you need such a tool, I would point you towards the NirSoft utilities). I dislike FUD and negative campaigning (sadly it works quite well).
A quick test in a VM showed it working as advertised. I have one remark though: while it doesn’t store the master password in memory (as the setup boldly points out), it stores the individual account passwords in clear-text when it is unlocked. You can check this out using something like Process Hacker or Process Explorer. You should point them to the actual browser process, since it is there where the passwords are stored, rather than the main RoboForm process.
One thing which Roboform doesn’t do is to differentiate between the HTTP and HTTPS version of the sites. It happily fills the secure version of the page with credentials saved on the insecure version and vice-versa, potentially exposing the user to and sslstrip type of an attack, which becomes more and more important because of the wide use of wireless technology.
Finally, I would like to comment on something Derek wrote in his post (so this is not an official statement from Roboform as far as I know): “ROBOFORM … keeps all passwords in a secure encrypted database that only you (not a keylogger or malware) can access and use it”. This is and isn’t true. It is true that keyloggers won’t see your individual passwords, but it will see your master password (!!!). Also, while Roboform will protect you against malware which specifically targets the password store of browsers (and there are quite a few of those out there), it will not protect you against the ones which inject themselves in the browser and simply capture the contents of any HTML forms which have an INPUT element of type PASSWORD in them - of course neither will SSL/TLS. And there are quite a few out there which do this.

In conclusion: I wouldn’t trust my sensitive passwords to a closed source program and I would always go with an open-source alternative. Roboform isn’t vastly superior in any particular way and the marketing around it leaves a sour taste in my mouth.

BTW, the links on Derek’s site are affiliate links - which is all nice and good, I too have affiliate links in my blog postings occasionally - but I would have liked a clear disclosure about this fact.

Who feeds me? (with information)

Attila-Mihaly Balazs — Mon, 25 Jan 2010 11:54:00 +0200

I received a private request to share the feeds which I read, an I though to do it publicly, since there might be other people interested in it (yeah, right :-p). Also, the least I can do is to post a link back to the authors website. So grab my OPML file or see the complete list below. Disclaimer: the presence of a given site in the list below shouldn’t be seen as an endorsement. I read a couple of feeds to “hear the other party”. Also there are a couple of dead feeds in were not cleaned up. So in no particular ordere here are the feeds:

blog.avast.com

000webhost hosting scam

Catherine: pyOraGeek

cktricky and Web Application Security

Comments on: yLife: Open database for Yu-Gi-Oh duels proprietary alternatives

Deep Rants

Digital Forensics Journal

Domber’s Basecamp

DonationCoder.com Radio Show Podcast

ESPM 160AC: American Environmental and Cultural History

Evilcodecave

Featured’s Screencast.com feed

feed/http://www.101webdesignshow.com/xml/feedmp3.xml

Firefox Power

flytecast: web strategies for small business

Fresh Voices - powered by PodTech

Freshtopia.net

Friends in Tech

Friends Of The Fringe

FS651 BoilerCast!

Fuming Incense Stencher

Geek Pit

Geekdrome MP3 audio

Geo 10: World Regions, Peoples, and States

George Ou

Global Nerdy

Google App Engine Blog

Hak5 Rainbow Tables

Help Desk Talk

History 5: European Civilization from the Renaissance to the Present

Humanized Weblog

IDS 110: Introduction to Computers

igxglobal Daily Security Briefing

InDigital

Indy News

Information Architecture (IA) Podcast Show Notes

InfoSec Jobs

InfoSys 296A-2 / Law276.8: Open Source Development and Distribution of Digital Information: …

InfoTalk - powered by PodTech

Intel - powered by PodTech

Intel Software Network RSS feed for Podcasts

Internet2 Network Upgrade

Juniper Networks - powered by PodTech

Juniper Networks J-News - powered by PodTech

Justin & Lindsey’s Adoption Story

Kevin’s eLearning Blog

Koral Blog

Less Than 3

lifehack.org

Limited-Exposure :: Networks, Security and Technology | News, Reviews, and Opinions

Linux on the desktop

Local Area Security

Maeda’s SIMPLICITY

Magnus Hagander’s PostgreSQL Blog

MainFunction::Editor’s Corner

Marketing Voices - powered by PodTech

Mike Christensen: Web Dev Guy

MSDN Magazine - Cutting Edge

msdn’s Showtime

Mubix’s Links

muzikDEN h.264

NerdTV - MP3 Podcast | PBS

NETGEAR - powered by PodTech

Network Consulting Services

Network General Sniffer Cast - powered by PodTech

Paul Thurrott’s Internet Nexus

PHIL103 Introduction to Philosophy

PHPR476 BoilerCast!

PLANET WEBSECURITY

PodSummit - powered by PodTech

PodTech News - powered by PodTech

POL101 BoilerCast!

Polycom - powered by PodTech

PolySci 179: Undergraduate Colloquium on Political Science

PowerUser.TV (Power User Podcast)

Robert Schmelzer is Thinking about IT

SABAGsecurity

Scott Stanfield

Seagate Backup Awareness Month - powered by PodTech

Search Engine Optimization(SEO) Mr SEO’s Podcast

Secure IT Live

SecureBlog

Security and Secure IT

Security Renaissance

Security Wire Weekly

Security-Protocols

Sick of spam? Get mad and get even: Blog

SOC100 BoilerCast!

SOC100T BoilerCast!

SOC312 BoilerCast!

Software, Technology, and Science

Spam chongqing

SPCH101 Oral Communication in Contemporary Society

SploitCast

Stat 2: Introduction to Statistics

Steven Sinofsky’s Microsoft TechTalk

Stranger Things - iPod (640×480)

SWTC CourseCasts: 806-177-01 A&P;

Symantec - powered by PodTech

Symantec Security Response Podcast

Tales From Boxcat Junction

Tasty Research

Technobabylon

the Accidental Creative

The Bleeding Edge

The ColdFusion Podcast

The Command Line

The Cubicle Escape Pod

The Cubicle Escape Pod Blog

The Digital Standard

The Dilbert Blog

The Front Line Trendcast for Medium Business: Medium Business

The Hacker’s Choice - News

The Laws of Simplicity

The Medium Bagel

The Official Parallels Virtualization Blog

The Podcast Academy

The Register - Software: Open Season

The Security Hookup Vodcast

the show with zefrank

trimMail’s Email Battles

Unplugged

Virtual War Unlimited

Web Design & Development Podcast by Engines of Creation.com

Web Standards Group PodCasts

webcast.berkeley: UC Berkeley Events | All

Webstock - News

::Planet PostgreSQL::

Bad CTK

blog

Brian “Krow” Aker’s Idle Thoughts

CMD: Joshua Drake’s Blog

Database Geek Blog

Databases and Performance

Denish Patel

Devrim’s PostgreSQL Diary

Diamond Notes

DimitriK’s (dim) Weblog

domas mituzas: vaporware, inc.

EnterpriseDB News

Everything is Data

Frakkle

Fujitsu Supported PostgreSQL blog

Greg’s Postgres stuff

I’m just a simple DBA on a complex production system

Just a Theory

Magnus Hagander’s PostgreSQL blog

Monty says

MySQL Performance Blog

PGSQL_Announce - TwitFeeder

Postgres Database News

Aaron Margosis’ WebLog

Absoblogginlutely!

Adam M Craven’s Technical Blog

Advice and Opinion - Executives Online

Anchor Web Hosting Blog

Andy ITGuy

Apache Infrastructure Team

Ed Bott’s Windows Expertise

Everything Sysadmin

Friends in Tech

George Ou’s technology blog

Grand Stream Dreams

IT Law in Ireland

Justin James’ Critical Thinking

obijuan: [techno]plebe

PacketLife.net Blog

Personal website of Dave Hope

Random things in IT

Retro Computing

Righteous IT

Roger’s Information Security Blog

Security Blogging under a mask

Systems Engineering and RDBMS

There is a small blog here.

A Couple Of Admins Podcasting

Adventures in Security

An Information Security Place

AudioParasitics - The Official Podcast of McAfee Avert Labs

Binary Revolution Radio

Black Hat Briefings, USA 2007 [Audio] Presentations from the security conference.

Blue Box: The VoIP Security Podcast

Brad Pollard’s 140 Seconds

brucon podcast

bsdtalk

Buzz Out Loud from CNET

Career Tools

CastaBlasta

Casting from the Server Room

Cloud Security Podcast

CNET News.com: Security Bites

ColdFusion Weekly

ComodoVision

CyberSpeak Podcast

Daily Internet Storm Center Threat Update

Dan Bricklin’s Log Podcast

dConstruct 2007

Dr. Kiki’s Science Hour

ESET Podcasts

Eurotrash Security Podcast

FRONTLINE - Reports | PBS

Hanselminutes

In the Trenches

Internet Storm Center Threat Update

Johns Hopkins University Podcast - Exploring Liberal Arts

Jupiter Broadcasting

Justice Talking

LugRadio (high-quality mp3)

Manager Tools

Marketplace - Marketplace Whiteboard

Millahseconds

Mondays

Network Security Blog

Network World’s Twisted Pair

Open Voices: The Linux Foundation Podcast

OurSQL: The MySQL Database Podcast for the Community, by the Community

OWASP Security Podcast

Pandora Podcast Series

PaulDotCom’s Web Site

Perlcast

php|architect’s Pro PHP Podcast

Ruby on Rails Podcast

Software Engineering Radio

The Algorithmic Trading Podcast

The HTML Show

The IT Security Pubcast

The Java Posse

The Linux Action Show! Podcast

The Reality Check Security Podcast

The Rear Guard

The Security Roundtable

The Silver Bullet Security Podcast

The Software Freedom Law Show

The Southern Fried Security Podcast

The Thirsty Developer - Podcast

The Web 2.0 Show

The WordPress Community

This Week in Startups - Audio Only

this WEEK in TECH - MP3 Edition

TPN :: The Cranky Middle Manager Show

TWAT Radio

Ubuntu UK Podcast » mp3-low

Uploads by pennsays

UXpod - User Experience Podcast

Valid Syntax

Warwick Podcasts

WeatherBrains

Web Axe - Accessibility Tips

Web Design Advice (boagworld.com)

WebDevRadio Podcast home - web development discussion

Welcome to the TechPodZone!!!!

Wie war dein Tag, Liebling? Mit Anke Engelke und SWR3-Moderator Kristian Thees

Wikipedia Weekly

WinDirStat weblog

Windows Weekly with Paul Thurrott

Zend Developer Zone | php_abstract

Fredo and Pidjin. The Evil Comic.

Kevin and Kell

Looking For Group

Nerdcore: The Core Wars

The 5th Wave - goComics.com

Ubersoft.net: Technology Is Not Your Friend

Userfriendly With images

Fun with virtualization

rakeshm’s VM Management Blog

Robert Larson

The Virtual Data Center

Virtual PC Guy’s WebLog

Virtual Scoop - latest scoop on virtual machine technology

VMware Security Blog

Windows Virtual PC

Windows Virtualization Team Blog

Dare Obasanjo aka Carnage4Life

Davis Freeberg’s Digital Connection

deconcept

delicious blog

DOM Scripting Blog

Elsewhere on the ‘Net

Extra Pepperoni

Firebug - Web Development Evolved

ginger’s thoughts

Google Analytics Blog

Google Custom Search

Google Enterprise Blog

Google Operating System

Google Website Optimizer Blog

Kalimat al-Mutafalsif

Matt Cutts: Gadgets, Google, and SEO

mnot’s Web log

molly.com

Niche blogs with Google AdSense

No Such Weblog

Official Google Blog

Official Google Reader Blog

Official Google Webmaster Central Blog

The Official FireHost Blog - Secure Web Hosting

The Opera Rootstore

Videos uploaded by websiteoptimizer

Vitamin

Web Standards with Imagination

Web Things, by Mark Baker

www.seoidiot.com

- biht -

-: A Random View of an Insecure World :

…And you will know me by the trail of bits

…And You Will Know me by the Trail of Bits

.:Computer Defense:.

0Kn0ck’s Blog - dt sns!_*

A for-now blog of Araz Samadi

A Spamtracker’s Blog

abuse.ch

ADD / XOR / ROL

Adobe Product Security Incident Response Team (PSIRT)

Al Iverson’s DNSBL Resource

Alex Ionescu’s Blog

Alex’s Corner…..

Ambersail Infosec Roundup

Amir Lev’s blog

Andrew Martin

Anti Rootkit Blog

Anton Chuvakin Personal Blog

Application & Threat Research Center

Artists Against 419

asmatiks’ zone

Attack Research

Australian Honeynet Project - Australian Honeynet Project

BjOG - Bjou’s Blog, that is!

BlueHat Security Briefings

botnetz.com

Bradley Schatz’ Weblog

Breaking Code

BreakingPoint Labs Blog

C skills

C.I.S.R.T.

call dwerd ptr [6c756973]

Captain’s Blog

Carlos Perez’s Blog

carnal0wnage.attackresearch.com

Christian Seifert’s Blog

christian’s weblog

Christophe Pradier on Security

Cisco Security Blog

Clerkendweller : Web Security, Usability and Design

Cloud Security Alliance Blog

Coding Insecurity

Columnist Guild Blog

Commtouch Café

Community Crisis Response Teams

Computer and Network Security, Mamak Style

Computer Forensics, Malware Analysis & Digital Investigations

Computer Security Is My Interest!

CyberCrime & Doing Time

Daemon on Security

Damn Spam!

Dancho Danchev - Mind Streams of Information Security Knowledge

Dasient Blog

Datasets for the Research Community

Dave Piscitello’s Personal Web Log

dave_rel1k - TwitFeeder

Digital Forensics and more

Digital Soapbox - Information Security, Risk & Data Protection Blog

DISOG

DK ‘Log

Donkey On A Waffle

Don’t Stuff Beans Up Your Nose!

eEye Digital Security - Zero-Day Tracker

eEye Research

Electric Alchemy

EM_386

Episteme: Belief. Knowledge. Wisdom

Errata Security

Everlasting Wanderer

Evil Fingers - The Blog

EvilFingers

Exploit Prevention Labs

Extended Subset

extern blog SensePost;

extraexploit

Extreme Security — Do It Securely or Not at all !

Facebook Application Smashing

Farfromr00tin

Fast Horizon

Financial Cryptography

Finjan MCRC Blog: Posts

Fitsec - Information Security Blog

Fix Winfixer!

Forage Security Inc.

forensic . seccure . net

fred avolio’s musings

Fred Avolio’s Musings

Gimme Hardware/Software Interface.

GlasBlog

GNUCITIZEN Media Portfolio

Google Online Security Blog

Greg Martin’s blog - InfoSecurity 2.0

Gustavo Duarte

gynvael.coldwind//vx.log (en)

haxorthematrix - TwitFeeder

Hex blog

HEXALE (security. reverse engineering. stuff.)

HexEsec

HiR Information Report

IBM Internet Security Systems Frequency X Blog

IBM Rational Application Security Insider

iDefense Labs Software Releases

idleloop

Iljas Blag

IncrediBILL’s Random Rants

Indefinite Studies

Infinite second

Information Centric Security

Information Security is not an Oxymoron

Information Warfare Monitor

(IN)SECURE Magazine Notifications RSS

Irongeek’s Security Site

Jeff Jones Security Blog

Joshua “Jabra” Abraham

Larry Osterman’s WebLog

Latest Alerts From Websense Security Labs

Latest web hacking incidents

Laurent Gaffié blog

Light Blue Touchpaper

LiveAmmo Computer Security News

Malware Analysis InDepth (MAI)

Marcus Ranum, computer security, photography, and other weirdness

Matt Blaze’s Exhaustive Search

Matthieu Suiche’s blog !

Michael on Security - Comments

Mischel Internet Security - Blog

MNIN Security Blog

Mozilla Security Blog

MSRC Ecosystem Strategy Team

Mu Dynamics Research Labs

Mubix Links

Musings of an Over-Grown Dwarf

Musings on Information Security

MW-Blog

mwcollect.org News

mxlab - all about anti virus and anti spam

myf00

Nart Villeneuve

Neil Carpenter’s Blog

Nenad Vijatov • 0x4e 0x56

Neohapsis Labs

Neohaxor.org

Netcraft

Network Forensics Puzzle Contest

Observations of digitally enlightened mind

ocean’s InsecLab

Off by On

Offensive Computing - Community Malicious code research and analysis

Offensive Security Blog

Old McDonald’s Farm

Omnivora

Open Information Security Foundation

OpenDNS Blog

OpenRCE: Articles

OpenRCE: Site Updates

OSF Data Loss Database Blog

oxff’s Blog

p42 labs

Pablo Rincon

Paranoid Linux Ninja Geek

pastebin.fail

Pat’s Daily Grind

PaulDotCom Community Blog

Pentoo’s blog

PeskyMalware.info Feed

philosecurity

Phn1x - Hamsterswheel

PHP Security Blog

PHPIDS » Web Application Security 2.0

PHX2600

Piotr Bania Chronicles :: http://blog.piotrbania.com

PlanAHeist.com

Planete Project

Plausible Deniability

Poromenos’ Stuff

Privacy, Security and UI Review

Professional IT Security Providers - Exposed

Programming stuff

Push the Red Button

RaDaJo (RAul, DAvid and JOrge) Security Blog

Rafal Los

Ramblings of the änal security guy

Ramblings++

Random stream of chars

Random Thoughts from Joel

RCE Cafe

Reading a Hacker’s Mind

REblog

Reiners’ Weblog

Renesys Blog

Report Security Flaws

Reusable Security

Reverse Engineering b10g | REM

ReverseEngineering: what’s new online

Reversing folies

Reversing It Out!

Ribadeo Hack Lab - Articles

Richard WM Jones

Ride The Lightning

Ring3 Circus

RioSec - Security WebLog

RLR-UK

Robert Hensing’s Blog

Roger’s Security Blog

Rogue Antispyware

Room362.com

root labs rdist

Rooted Your {0x2E} Com

ryanlrussell

S!Ri.URZ

s3c-watch

SANS Internet Storm Center, InfoCON: green

Secdev - Thierry Zoller

SecSci Social Scene

SecTech

SecTechno

Secure Computing: Sec-C

Secure Home Networks

SecureThoughts.com

SecureWorks Research Blog

Security for Canadian Developers

Security Garden

Security Insights Blog

Security Is Simple: Only Use Perfect Software

Security Mike’s Blog

Security Ninja

Security Onion

Security Product Testing

Security Research by Alexander Sotirov

Security Retentive

Security Ripcord

Security Sauce

Security Second Thoughts

Security Tips & Talk

Security to the Core | Arbor Networks Security Blog

Security Vulnerability Research & Defense

SecurityZone.org - Information Security Blog

Silviocesare’s Weblog

sirdarckcat

skeptikal.org

slight paranoia

Small Blue-Green Blog

SMBlog — Steve Bellovin’s Blog

SNOsoft Research Team

Software Vulnerability Exploitation Blog

Someone Else

souriz’s weblog

Spacequad AntiSpam Services

Spam in my inbox

Spamhuntress

SpamIt Must Fall

Speaking of Security, the RSA Blog and Podcast

Stacks of Shame - an epitaph of bad coding…

Stefan’s Computer Center

System Advancements at the Monastery

System Integrity Team Blog

Taking Network Security to the Streets

Tales from the Crypto

Techie working in a corporate world

Technicalinfo.net Blog

Telic Thoughts

terminal23

Terry Zink’s Anti-spam Blog

The Antivirus Guy Blog

The Art Of Noh

The Dark Visitor

The Day Before Zero

The Ethical Hacker Network RSS News Feed

The Final Stream

The FORWARD project blog

The Guerilla CISO

The Honey Stick Project

The HP Security Laboratory Blog

the JoshMeister on Security

The Merchant Account Blog

the Month of PHP Bugs

The NoAH Blog

The Real Hustler

The SABRE Lablog

The Security Development Lifecycle

The Security Shoggoth

The Security Skeptic

The Spamhaus Project News Blog

The Spanner

The SpywareGuide Greynets Blog

Threat Chaos

Threat Researcher

ThreatExpert Blog

ThreatLevel - TwitFeeder

time to bleed by Joe Damato

Uncommon Sense Security

Under The Hood - Matt Pietrek

Unmask Parasites. Blog.

UploadMalware.com’s Malware Blog

Uploads by teamcymru

VERT

ViCheck Malware Trends

Web Security Blog by Purewire

Websecurity.ro

Websense Security Labs Blog

Welcome to the dregs of my mind

When {Puffy} Meets \^RedDevil\^

WhiteHat Security RSS Blog Feed

Will Hack For SUSHI

Windows Incident Response

Windows Live OneCare Team Blog

Wintercore

winternals: Windows OS internals and programming

Yet Another Infosec Blog

Zaphod’s Web Wanderings

Linux.com :: Features

Mostly Linux

My life’s journey

shotofjaq - TwitFeeder

The Linux Society - The AberTayLUG

Ubuntu Linux Blog by Ralph

Yet Another Linux Blog

Anti-Malware Engineering Team

anti-virus rants

Application Control and Device Control for Windows Desktops

Assa’s blog

Authentium Virus Blog

AV-Comparatives weblog

avast! blog

Avira - TechBlog

Blog VirusTotal

CA Security Advisor Research Blog

Chester Wisniewski’s Blog

Clamwin RSS Feed

Countermeasures

eSage Lab company news

F-Secure Antivirus Research Weblog

F-Secure Linux weblog

FireEye Malware Intelligence Lab

Forefront Client Security Team Blog

Fortinet FortiGuard Blog

McAfee Avert Labs Blog

McAfee SiteAdvisor Blog

Microsoft® Malware Protection Center

MoMusings

NanoScan Blog

News from the Lab

Norton Protection Blog

Security Response Weblog

SophosLabs blog

SunbeltBLOG

Symantec Connect - Security - Blog Entries

The AVIEN Blog

ThreatBlog

ThreatFire Research Blog

TREND MICRO Security Blog

TrendLabs | Malware Blog - by Trend Micro

Virus Lab

Welcome to AMTSO - The Anti Malware Testing Standards Organization

Zª╔εs±é└

ASCII by Jason Scott

bin-false.org

blog.reddit — what’s new on reddit

Bob Frankston’s Writings

Freakonomics

Google Logos

Grooveshark

grooveshark - TwitFeeder

I Heard It On NPR

Isotopia - Variations, on a theme
Home of Xenon’s Stuff

Johannes Koelman’s blog

Lunduke.com

Not Always Right | Funny & Stupid Customer Quotes

Rosswriting

The Pathos Daily

The Quantified Self

The “Blog” of “Unnecessary” Quotation Marks

Things That … Make You Go Hmm

Andreas Gohr: Weblog [splitbrain.org]

ch!mer!c.de

CH!MER!C.de

ch!mer!c.de blog

DokuWiki at Yahoo! Groups

Blog Entries

FeedBulletin for: sandrosaitta

Life Analytics

Neural Market Trends

2007 Perl Advent Calendar

2008 Perl Advent Calendar

2009 Perl Advent Calendar

Audrey

Dev411 Blog

Modern Perl Books, a Modern Perl Blog

Nordic Perl Workshop 2009

Not this…

Parrot

Perl 6 Advent Calendar

perloneliner - TwitFeeder

Shlomi Fish’s Journal

Google Alerts - cd-man

Google Alerts - hype-free

hype-free

Software Engineering Radio - Comments

8 bits

A Computer Scientist in a Business School

Atlassian Developer Blog

Breaking Eggs And Making Omelettes

Chris Jackson’s Semantic Consonance

Cipher Prime Blog

Cliff Click Jr.’s Blog

DebugInfo.com - Oleg Starodumov

dive into mark

Doboism

Encytemedia - Home

Eric Fortier’s Bridging The Gap

foobar: foobar on computers, software and the rest of the world

GLOG

good coders code, great reuse

Google Code - Updates

Google Open Source Blog

hackety org

HackMii

Hardwarebug

Hexadeciman - 16-bit Programming Blog

Humboldt Solutions

I. M. Wright’s “Hard Code”

If broken it is, fix it you should

it’s all guid

jCraze Blog

JEDI Windows API

JeffCroft.com Journal

Jensen Harris: An Office User Interface Blog

Jeremy Zawodny’s blog

Johann Burkard

Kernel Mustard

Krugle Blog

Laurentiu Cristofor’s blog

Manuel Aldana

MaraDNS

Mechanix

Microsoft Office 2010 Engineering

mishou.org

Mixxx Development Blog

Monsters Got My .NET

Moserware

Murray Sargent: Math in Office

Nadia Alramli’s Blog

NASA NEBULA News

NDIS Musings

Nektra Advanced Computing Blog

Neopythonic

Net::DNS maintenance Blog

Otaku, Cedric’s weblog

Paint.NET

Personal website of Dave Hope

RED news and information

RemkoWeijnen.nl

Roads Less Taken

Scott Hanselman’s Computer Zen

slayeroffice.com - web experiments gone horribly awry

Squawks of the Parrot

stackoverflow

steike.com

Stevey’s Blog Rants

Suckbusters! from David Platt

Tales of a Code Monkey

tenshu.net

Terminal Services Team Blog

The Audio Fool

The Braidy Tester

The Data Compression News Blog

The GitHub Blog

The History of Python

The Old New Thing

The Oracle at Delphi

The PowerPoint Team Blog

There’s not a moment to lose!

Windbg by Volker von Einem

WinDirStat weblog

Windows Insight Blog

WinDrvr

About Romanian Web Security

Adi Roiban

AlexJ

asdf vs hjkl

BalaBit Corporate Blog

BalaBit IT Security Blog

Bani pe net - by Cotiso

Comentarii pentru: Startup JUG Cluj Napoca

Convertor.ro

diacritica

edy si lory

Grupul pentru software liber

JAVA User Group Cluj-Napoca, Romania

Jurnal de noapte

La Andu

Life is life [Tudor Salomie’s Blog]

Transylvania JUG Google Group

transylvania.java.user.group’s Photo Gallery

tudy .ro - Tudor Damian

Twindows Internals

Twitter / vtopan

UberGeek

Webdevelopment with Style

MintLife Blog | Personal Finance News & Advice

Mish’s Global Economic Trend Analysis

naked capitalism

neuralmarket - TwitFeeder

Rortybomb

The Market Ticker

Tim Harford

zero hedge - on a long enough timeline, the survival rate for everyone drops to zero

Splitting hairs^H^H^H^H^H strings with Java

Attila-Mihaly Balazs — Sun, 24 Jan 2010 21:55:00 +0200

[![Split Personality](http://farm3.static.flickr.com/2097/2225546021_4864c2bc72.jpg)](http://www.flickr.com/photos/nickwheeleroz/2225546021/ “Split Personality by nickwheeleroz, on Flickr”)

Offtopic: where does \^H come from? (since I too found it only recently) - from the source of all wisdom - Wikipedia :-p

Pressing the backspace key on a computer terminal would generate the ASCII code 08, BS or Backspace, which would delete the preceding character. That control code could also be accessed by pressing Control-H, as H is the eighth letter of the Latin alphabet. Terminals which did not have the backspace code mapped to the function of moving the cursor backwards and deleting the preceding character would display the symbols \^H (caret, H — see Caret notation) when the backspace key was pressed. This sequence is still used humorously for epanorthosis by computer literates, denoting the deletion of a pretended blunder, much like overstriking.

Now back to our topic: how to split a String in Java into parts? Here are a couple of options:

Use a StringTokenizer - one thing which surprises Java novices is that the second parameter in the constructor is a list of separators not one big separator. So if you specify ", ", this will split at either commas or spaces, not at commas followed by spaces.
Use String.split - this can also surprise novices since - even though it accepts a String parameter - the String will be compiled into regular expression (Pattern). While there are several warning signs (like the documentation - because everyone reads those :-p - or the parameter name), it is easy to ignore and one can get surprising results (like splitting the String "aaa.bbb.ccc" using the pattern "." and getting an empty String).
Precompiling a Pattern and using the split method on it. This has the advantage of expressing intent more clearly than the previous method and also being more efficient if one needs to split multiple strings using the same Pattern. However this still can be overkill if you just want to split using a single character (or even a list of characters).
Using the StringUtils.split method from the Apache Commons library (which can be used in commercial projects!) or even roll your own.

When considering these alternatives you should consider expressiveness first and performance second (also, don’t optimize until a profiler tells you to!). Luckily StringUtils is a clear choice from both points of view, unless you need something more complex than one character separators. Here are some performance numbers:

Custom implementation (Apache Commons should be in the same ballpark) - 1
StringTokenizer - 2.1
String.split - 3.46
Pattern.split - 2.87

These numbers shouldn’t be taken as an absolute and they can depend on many things (for example if a profiler is or isn’t attached, the version of the JVM, etc), but they should point you in the right direction.

Update: a very important note: while these solutions are mostly drop-in replaceable, be aware that there are some differences between them around the corner cases (ie. how they handle null’s, what if the string ends in a delimiter, what if two delimiters follow each other, etc). If and when you decide to swap one for the other, be sure to unittest your piece of code with the types of data you expect it to handle.

So there you have it, more than you ever wanted to know about splitting Strings :-) Hopefully this helps somebody.

The future of web hosting

Attila-Mihaly Balazs — Sun, 24 Jan 2010 20:04:00 +0200

I find myself thinking more and more about (a possible) future of the web.
While the current system for web hosting works pretty well (and I am the first
to say that we shouldn’t make the perfect an enemy of the good), it would be
really nice to exploit more of the potential in the web. What we could do
(and we already have the technology!):

Make as many aspects of it fully distributed as possible: this would mean
both content lookup (discovery) and hosting: today the average PC uses very
little of its resources (whether we are talking about storage, bandwidth or CPU).
We could push out content to the “edges” which would result in a better utilization
of the local network infrastructure (hence ISP would gain from it) and faster
content delivery without compromising secrecy if we so desire using PKI (hence
content producers should like it).

As a “sideeffect” we could get an information system which has integrity built in
by default (ie. X said Y at moment T - certified by cryptographic protocols) and
is highly resistant to censorship (because information is distributed and encrypted
by default both in transit and at rest). What a nice place it would be!

Full disclosure: this is a paid review from ReviewMe. Under the terms of the understanding I was not obligated to skew my viewpoint in any way (ie. only post positive facts).

Best web hosting?

Attila-Mihaly Balazs — Fri, 22 Jan 2010 22:23:00 +0200

There is a lot of talk out there about which is the “Best Web Hosting” company.
With such a highly contentious subject (since being the “best” means more
customers - and implicitly more money), there are a lot of “authoritative sources”
out there. One authoritative (without quotes!) source would be Netcraft, but
then again, their statistics are somewhat limited in this space - it is nice
to know who were the fastest or who had the smallest downtime, but this doesn’t
tell me anything about pricing, customer support and other services (for example
DDoS resistance/mitigation can be quite important these days!).

So, how to choose? I have just two words - very carefully. Have a clear goal in
mind (ie. “I want a CMS” or “I want a blog”) and try not to overestimate your
needs (especially if it is a personal project). The most important thing is to
own a domain (yeah, I known, look who’s talking) - which you can redirect later
on if you decide to move providers. I’m a great fan of niche/dedicate hosting
(like Blogger, Wordpress, or why not - Squarespace). Given their specialization
they can deliver their core competency very efficiently and securely and very
often free of for a low cost. The second best thing would be a (managed) VPS
and only in the last instance would I consider shared hosting.

Full disclosure: this is a paid review from ReviewMe. Under the terms of the understanding I was not obligated to skew my viewpoint in any way (ie. only post positive facts).

Donating to the Haiti earthquake relief effort

Attila-Mihaly Balazs — Mon, 18 Jan 2010 15:04:00 +0200

Cross-posted from the TwitFeeder blog. You might also want to check out the Options for giving post, which enumerates more organizations you should consider donating to (not related to the Haiti earthquake).

As most of you probably already know, the Republic of Haiti has been hit by a catastrophic earthquake. Please donate to the organizations which are trying to help. The online donation has been proven a great success - so good to see the bright side of globalization (speaking of online - it is an interesting example of how offline events can be observed online - an other example being the Google flu-trends). Below you will find a compilation of reputable information sources, which is important, since there are always a lot of scams around important events like this.

Important! Most of the information is rather specific to the USA and most of them presuppose credit-cards. If you live outside of North-America and have PayPal account, I would recommend donating to the Haiti Earthquake Children in Emergency Fund. You can get a list of ways you can donate via PayPal/eBay on their blog.

Doing the Right Thing from the SANS diary - it gives a couple of very good advices. It also specifically lists the following organizations:
If you have a Google Checkout account, you can donate to CARE or UNICEF using it. The Google crisis response webpage also lists a lot of trustworthy organizations which accept your donation and SMS numbers (valid in the USA only!) which can be used for donation. They also offer free calls to Haiti trough Google Voice
Here are a couple of great tips from the CERIAS blog (written by professor Spafford) to help you avoid being scammed:
- Do not enter any information at a Web page that pops up unexpectedly when you visit some other site.
- Never click on a Web site address (URL) in e-mail sent to you; it may look official, but most will be pointers to fraud or attack sites.
- Don’t assume that every Web address returned by a search engine, such as Google or Bing, is a legitimate organization.
- Do not respond to e-mail requesting donations or making a special offer (such as asking you to hold their assets while Haiti rebuilds).
- Do not reveal any personal or financial information during a phone call you did not dial yourself.
- If a friend forwards a URL, phone number or e-mail, don’t trust it until you check its validity. Your friend may have been scammed first.
A similar article on the Microsoft website: How to avoid online donation scams
If you are from Canada, here are a couple of resources (via Security Balance):
```
RED CROSS: www.redcross.ca
WORLD VISION CANADA: www.worldvision.ca
UNICEF: www.unicef.ca
SALVATION ARMY: www.salvationarmy.ca
MÉDECINS SANS FRONTIÈRES: www.msf.ca
```
- Many community websites also include a suggestion to donate (for example reddit.com or oneforty) and you can be reasonably sure about their recommendation if you’ve been using the site for some time (then again, it never hurts to double-check).

Picture taken from rbrwr’s photostream with permission.

An useful Ubuntu (online) resource

Attila-Mihaly Balazs — Mon, 11 Jan 2010 22:28:00 +0200

![](https://wiki.ubuntu.com/UbuntuMagazine?action=AttachFile&do=get&target=fclogo.png)

If you use Ubuntu, the full circle magazine is a great resource, go check it out. For example from the latest issue I’ve learned that I should install vim-nox on my servers if I want to upgrade my 1980’s experience to 1990’s :-).

I there is one (small) issue, it is the fact that the PDF is very CPU intensive (I’m not really sure about the cause), but it is very good looking and professionally done. Go look at it if you use Ubuntu. And consider donating or writing an article or two :-)

Forensic analysis of JPEG images

Attila-Mihaly Balazs — Mon, 11 Jan 2010 18:07:00 +0200

Recently I became aware of the Hackerfactor blog, especially the posts related to discovering image manipulation. It is interesting to read what one can deduce from an image, even when one doesn’t use such “obvious” information sources like image metadata (I say “obvious” because it seems that it isn’t obvious at all for most people – but at least it can sanitized automatically). So here are the links to the tools he recommends:

You might also find this paper interesting.

All in all, the most interesting thing for me was the fact professional image manipulators (ok, I just made that word up, meaning “people who know keyboard shortcuts in Photoshop”) repeatedly re-save the same image in lossy formats like JPEG, thus compounding the loss of quality. Then again, one should never underestimate human stupidity.

Picture taken from Elsie esq.’s photostream with permission.

Security vendor’s “top-threat” list proof for their less-than-perfect performance?

Attila-Mihaly Balazs — Mon, 11 Jan 2010 17:52:00 +0200

Here is something I’ve been thinking about lately: most (all?) security vendors publish their “top-threats” periodically. Those lists are made up by centralizing numbers reported by their clients. While it is safe to assume that the majority of the enumerated threats are blocked straight-away – before they can execute a single piece of code – there is a certain percentage which is after-the-fact detection (ie. the machine gets infected, a signature comes out later on at which point – if you’re lucky – the security program will block the malware).

Now I have no idea about the relative size of this subset (or if the companies have it, or how they can collect it for that matter), but I find the idea that marketing material put “out there” can backfire amusing :-).

Picture taken from tigger1fic’s photostream with permission.

A missed opportunity

Attila-Mihaly Balazs — Fri, 08 Jan 2010 17:51:00 +0200

The theory of capitalism (and I’m greatly oversimplifying here, I know) says that, even is we all follow just our own self interest, a global “good” will somehow emerge. This is what F-Secure is doing in their blogpost where they write about a specific ransomware which – if you get infected with - encrypts your data and asks you a certain amount of money to decrypt it.

Trouble is that their only recommendation is to “remind everyone to backup their important files regularly” (coincidentally – sarcasm, sarcasm – they have an online backup component in their suite). They could have at least mentioned that Sunbelt provides a tool which may decrypt the files (I say may, because I didn’t actually try the tool). This is even more inexplicable given the fact that they got the samples from Sunbelt (“Many thanks to Adam Thomas from Sunbelt for providing samples of the dropper”).

Shame on you F-Secure for putting a (possible) financial interest before the interest of your users!

So I don’t know about you, but instead of claiming that pure self-interest is the solution, I will go with:

Everything in moderation - including moderation.

Picture taken from d3stiny_sm4sher’s photostream with permission.

PS. Who wants to bet that – if these claims are bought to F-Secure’s attention – they will claim that they didn’t know about the removal tool?

Update: I’m not singling out F-Secure here, Zarestel Ferrer from CA just made a very similar blogpost: here are the facts (he did include some more technical detail, which is nice for us, security geeks), you should have used a security product to keep it out:

CA advises to keep your security products signature updated to prevent this kind of ransomware.

The plus side: he doesn’t pimp his company’s product necessarily. The minus: he doesn’t link to the Sunbelt decryption tool either. On the plus side, there is a comment facility on their website which could be used by visitors to mention the tool and thus help out people who lost data, but on the negative side: it doesn’t work, not even with IE!.

Announcing a couple of contests

Attila-Mihaly Balazs — Thu, 07 Jan 2010 19:22:00 +0200

Here are some contests I found:

The Fifth Underhanded C Contest – the scope is to write benign looking code which would pass trough a code-review (and/or there is plausible deniability for the coder), but which does some evil things. Found it via slashdot. Some of the previous solutions are truly ingenious.
From the Forensics Contest comes Ann’s AppleTV, where you can win … Ann’s AppleTV of course!
Forensic Practical Exercise #3 – no prizes, just fun (and it isn’t all that simple either!). If you don’t own encase, you can transform the provided image into a DD style raw image using the Sleuth Kit:
```
img_cat.exe -v -i ewf Forensic_Practical_3.E01 > dd.raw
```
Just consider that the resulting image will be around 4G (because the thumb drive it imaged was 4G).

Good luck to everyone!

PS. The current Ethical Hacker Challenge is still open until the 11th of January.

Options for giving

Attila-Mihaly Balazs — Thu, 07 Jan 2010 19:12:00 +0200

I won’t go as far as to say that the calendar system is arbitrary and there is no reason to follow it :-p, but I have to say that giving donations shouldn’t be limited to Christmas. So if are/get in a giving mood, here are some geeky organizations you could donate to:

There is an auction going on on eBay for a very rare userfriendly.org T-shirt. All the proceedings will be donated to cancer research (there is 1 day and 8 hours remaining as of this moment)
Buy Bad Code Offsets. 100% of the money will be donated to open source projects
Most of the “free” services on the Internet accept donations: go donate to your favorite open-source project / podcast.

When we share, we all get richer. (ok, this is not mathematically exact, but it sure sounds nice, doesn’t it?)

Update: here are a couple more projects/organizations you should consider donating to:

The EFF - Electronic Frontier Foundation: “the leading civil liberties group defending your rights in the digital world.”
The SFLC - The Software Freedom Law Center: “We provide legal representation and other law-related services to protect and advance Free, Libre and Open Source Software (FLOSS)”. Also, you should check out their podcast which is always full with interesting information.
Wikileaks: “The Sunshine Press (WikiLeaks) is an non-profit organization funded by human rights campaigners, investigative journalists, technologists and the general public. Through your support we have exposed significant injustice around the world—successfully fighting off over 100 legal attacks in the process.”
OpenStreetMap: “The Free Wiki World Map” AKA the community created and owned map of the entire world.

Picture taken from Mickael Casol’s photostream with permission.

Making Prodigy’s “Smack My Bitch Up” in Ableton by Jim Pavloff

Attila-Mihaly Balazs — Thu, 07 Jan 2010 18:59:00 +0200

I just remembered what a Prodigy fan I was. The old man :-p doesn’t like it, but that’s ok. I don’t like Tokyo Hotel either :-)

[youtube=http://www.youtube.com/watch?v=eU5Dn-WaElI] And here is even more Prodigy music, curtesy of Grooveshark:

http://listen.grooveshark.com/widget.swf

TwitFeeder is not for Twitter only

Attila-Mihaly Balazs — Wed, 06 Jan 2010 11:37:00 +0200

The TwitFeeder has support for non-Twitter sources - crossposted from the TwitFeeder blog:

The TwitFeeder uses the Twitter API to get the different bits and pieces of information needed to generate the enhanced RSS feed. However Twitter isn’t the only one exposing the API:

Laconi.ca (now Status.Net) is an “Open source microblogging service” and it exposes an API which is almost 100% compatible with the Twitter API and you can see it in action on Laconi.ca sites like Identi.ca or The TWiT Army (the Status.Net wiki also contains a large list of installs)

Wordpress.org recently added support for a subset of the Twitter API

What does this mean for users? All the Twitter specific tools just became more usable, leading to more integration and less time-wasting (which is the main idea behind this site). You can use the TwitFeeder to subscribe to any site which exposes the Twitter API:

Subscribe to one of Identi.ca’s founder: identi.ca/evan

Subscribe to Leo Laporte on the Twit Army: army.twit.tv/leolaporte

Subscribe to a Wordpress blog (most of the blogs already have an RSS feed, but just for the fun of it): wordpress.com/decipherinfosys

More API support = Less siloed information = More fun

Programmers sidenote: there is no “official” compatibility test for the API, so you can never be sure that you are perfectly compatible – there will be always some gray areas (for example the exact link to a status message – which varies between different implementations). All you can do is to have rigorous unit tests. And code reuse either using our sources or pything-twitter to which we’ve contributed support for multiple sources.

Google Web Hosting alternatives

Attila-Mihaly Balazs — Wed, 06 Jan 2010 10:25:00 +0200

Google is one of the “big three” integrated web hubs on the internet offering many services in one place, most of them free. Of course everything has it price (nothing is truly free) and these offerings being free means that they don’t guarantee a very end-user friendly SLA. For example they recently retired Google Pages and now offer Google Sites. While the services are similar (and Google tried to automatically migrate as much of the content as possible), there are some corner cases which were served by the former but not by the later (for example it is my understanding that you can’t create custom HTML / CSS / JS – or upload arbitrary files for that matter - on Google Sites).

The following blog post offers alternatives to the google web hosting (which would be going with a hosting company – either shared, VPS or dedicated; buying your own domain name and installing your CMS – or having someone install it for you). Why I still lean towards the Google offering:

their SLA is still better than most of the industry’s (and certainly better than you can achieve by running your own server!)
the limited feature set means that they can optimize the heck out of it by adding redundancy and load balancing, even across data centers – if your server/datacenter fails, you will have downtime. In contrast hardware failure for Google is a daily thing and the services are built to work around it
they gave lots of advance warnings that the change was going to happen

Full disclosure: this is a paid review from ReviewMe. Under the terms of the understanding I was not obligated to skew my viewpoint in any way (ie. only post positive facts).

A “Bob” story

Attila-Mihaly Balazs — Mon, 04 Jan 2010 18:06:00 +0200

If you are not familiar with the “Bob story” concept: I first heard about it on the Pauldotcom podcast, where Twitchy used to tell stories about how “Bob” went wardriving, created a fake AP and did other grayhat things. They may have taken the idea from somewhere, but this is where I’ve heard it first.

Before Christmas Bob decided to treat himself with a Wii Fit (yes, the only thing which came of it was a machine telling him that he is fat – don’t ask :-p). Looking around several places he finally decided to go with a supermarket, since it was the cheapest choice (by about 30 EUR). He picked it up and rushed trough checkout. After paying he realized that the anti-theft system wasn’t removed, but since it didn’t beep when exiting the store he figured that it should be ok.

So he rushes home and cuts one of the wires (sidenote: with very low-tech equipment it took less than two minutes to do that. Using something professional, yet still discreet enough to fit in a pocket, it would take less than 30 seconds). And behold: it started to beep. The beep was somewhat loud, but not so loud as to be heard outside of an apartment, especially when put in some kind of acoustic insulation. Most probably it would attract attention in a supermarket though. And thus the alarm system met ~~its maker~~ a hammer and was promptly silenced. Just for the fun of it, Bob decided to take it completely apart to see how it worked:

Ideas / lessons / thoughts:

The construction is not that robust and would quickly give way if a determined / knowledgeable attacker was picking away at it (physical force is optional)
It is powered by four small batteries. In fact, it is very easy to deactivate by using Philips a screwdriver and removing the batteries. In what Bob can only assume to be standard operating procedure, the battery access was oriented towards the box and became accessible only after the unit was removed from the box
It would be interesting to try the following experiment, cut the insulation in two places on one of the wires. Link together the two cuts with a good conductor (like a copper wire). Now cut it in the middle. If the system isn’t watching for fluctuations in the resistance, just for contact / no contact it should work (which is probably the case, since the alarm didn’t trigger until Bob fully cut trough the cable, even though he partially cut it before)
There are actually two pair of wires crossed rather than one long wire. The “normal” deactivation mechanism is probably a magnet which allows the case to be turned counter-clockwise (this is normally blocked by the combination of metallic plate (upper side of the picture) and the white ring (which was “teeth” oriented in one direction). Crafting a small deactivation device seems also rather easy.

My conclusion is that (similar to infosec) “you don’t have to outrun the bear, only your friend” – meaning that you only have to raise the bar a little to eliminate most of the problems. Then again, one should go into such endeavors with “eyes open” and knowing the limitation of the technology (which very few vendors will tell you). And, as always, it will annoy your customers!

Launching The Twit Feeder

Attila-Mihaly Balazs — Sun, 03 Jan 2010 20:38:00 +0200

As the readers of the blog know, I’ve been slicing and dicing the Twitter RSS feed for some time. Finally I launched a free service which wraps all the improvements in one, easy to use and best of all, free (!) service: The Twit Feeder.

Cross-Posted from The Twit Feeder blog:

Efficiency is key for anyone consuming large amount of information. One of the keys to efficiency is to centralize your information consumption as much as possible. This is the reason why I chose to subscribe to the RSS feeds of the Twitter accounts I liked, instead of using the built-in follow mechanism: because I already use Google Reader and it is much less cumbersome to add an other feed into it, than to add an other site to my daily routine.

The Twitter RSS is a little lacking in features (understandably so, since it isn’t their main focus), so I’ve created a service which “enriches” the plain RSS feed:

@links and #hashtags are made clickable (example)

short URLs are expanded (example)

the avatar of the author is shown (example)

smileys are transformed into their pictorial representation

it is possible to subscribe for hashtag feeds, not just individual users (example)

geo-tagged status updates contain a link to the map showing the location (example)

So there you have it. Use it, enjoy it, support it.

PS. What should the next feature added be? I can think of two:

Create a badge (which can be embedded on third-party websites) showing the follower count and the subscriber count trough Twit Feeder

Make it possible to embed a HTML version of the feed on third party websites (like blogs)

Do you like these features? Vote for them by starring them. Have other ideas? Add it to our issue tracker. Also, bribes are welcome and they speed up the implementation of a feature considerably ;-)

Choosing a Java profiler

Attila-Mihaly Balazs — Sun, 03 Jan 2010 11:17:00 +0200

Recently I’ve been looking around for a Java profiler (since the two things you need for a successful performance tuning session are good data and clear targets). I’ll share the notes about my findings in the hope that they might be useful for someone. Quick disclaimer: don’t believe everything you read on the Internet! These are my own findings / experiences / opinions and they might not match yours. Also, they are specific to this point in time. It is quite possible that later versions of the given product fixes some / all of the problems I’ve experienced. An other distinct possibility is that I’ve overlooked something. If this is the case, please leave a comment and I’ll update the post ASAP.

Options I’ve eliminated completely:

AMD CodeSleuth - even though it is free as in freedom (open source), it has some serious shortcomings: it needs Windows (!) + Java 1.6 (might be a problem if you’re working with legacy apps) + Eclipse. Also, when I’ve tried to test it in a VM (VirtualBox running Windows XP on an Intel machine with VT-x enabled), it promptly bluescreend the machine (even though the documentation clearly says that you do not need an AMD processor to use it)
JRat%20) - FLOSS, but it doesn’t seem to be developed actively and has a limited set of features (but it worked, never skipping a beat, when I’ve tried it)
Eclipse TPTP (Test & Performance Tools Platform) - Eclipse specific. I’ve tried to install two different versions under Windows and - even though the installation seemed successful both times - it didn’t function as expected (in fact, whenever I tried to launch an application in profile mode, Eclipse just hung)
JProbe - Commercial, and it has a confusing licensing structure (I get easily confused - :-p)There are three components (Memory / Performance / Coverage Analysis) and each needs to be licensed separately…

I narrowed down my choices to three possibilities:

YourKit from YourKit LLC
JProfiler from ej-technologies
the NetBeans profiler (~~based on VisualVM~~ - I have been informed that in fact the reverse is true: VisualVM reuses the profiling code from NetBeans)

Things they have in common:

Multi-OS support (Windows / Linux / Mac / Solaris)
Multi-IDE support (IntelliJ, Eclipse, Netbeans) - with the exception of the NetBeans profiler, obviously :-)
Support for multiple / older versions of Java (again, important if you are working on a legacy project): 1.4 / 1.5 / 1.6 0 with the exception of the NetBeans profiler, ~~which is based on VisualVM~~ shares the profiling code with Visual VM, and - although I couldn’t find any explicit mention about this in the NetBeans documentation, the VisualVM site has a table which shows that profiling is not available for older versions of the JVM. Update: I have been informed (see the comment below) that these are not in fact limitations of the NetBeans IDE, only of VisualVM
Support for remote profiling (again, with the - possible - exception of NetBeans - the same situation as above)
Memory / CPU profiling
Thread state monitoring (runnable / running / blocked / waiting)
Dynamic instrumentation (no recompilation needed)
Saving and comparing of snapshots

CPU profiling overhead (measured by micro-benchmark which calculated PI with increasing precision):

YourKit: \~5x slowdown
JProfiler: \~10x slowdown (remark: JProfiler has a nice feature which suggests methods to exclude after the first run - ie. methods from the base libraries which are very frequently called - to reduce the profiling overhead)
NetBeans / VisualVM: \~4x

Remarks: these were “full profiling” results. Some profilers (JProfiler for example) support an alternative method: taking a look at the current stack for each threat at each N ms. This alternative method has a much smaller overhead and most of the time gives the same relative result (ie. the ranking of the most costly methods) even if the absolute times are not as accurate.

YourKit details:

http://www.yourkit.com/purchase/index.jsp#licensing_policy
It has the best integration with Eclipse from the three.
In the default setup it only shows the result after the application has (properly) ended. If the application terminates unexpectedly (for example you kill it), the results won’t be shown
It has no wizards for running user-specified Java programs (ie. from outside of the IDEs)
I’ve been informed that both of the previous shortcomings can be avoided by using the “remote” profiling feature (ie. by starting your program manually with the given agent). While this very well be true, it also sounds extremely cumbersome.
A possible advantage is that they also have a .NET profiler in addition to the Java profiler, so if your programmers regularly work in “both worlds”, or if you have distinct teams which do, you might be able to get a deal from them (I didn’t see anything on their site indicating that there was such an offer, but I imagine that it can be negotiated)

JProfiler details:

http://www.ej-technologies.com/buy/jprofiler/floating/volumeDiscounts?itemId=517127
Has a simple wizard for starting applications
It was the only one from the three which could show the stacktrace for the threat holding the lock, not just for the one waiting for it (very important, since it removes a lot of the guesswork!)
It was the only one (out of the three) which threated locks from the java.util.concurrent.locks package the same as “synchronized” blocks (both in the thread view - where it correctly displayed “blocked” instead of “waiting” - and in the “monitor” view). The other profilers knew only about “synchronized”

NetBeans details:

FLOSS (Free, Libre, Open source software)
It has a decent profiler ~~with all the essential functionalities based on VisualVM~~ - see the comment below
~~Unfortunately this means that it has the same limitations as the VisualVM technology: only local profiling on the 1.6 JVM is supported (this might or might not be a big deal for you)~~
I have been informed that NetBeans fully suppots older version of Java and also remote profiling scenarios.
One can use VisualVM directly, with the major advantage of being able to profile any local Java application in an ad-hoc manner (ie. the given application doesn’t even have to be started with a specific java agent, it is injected dynamically). There are of course some limitations (as with any technology): AFAIK, there are some limitations to the “visibility” of local Java applications (something along the lines of “has to be started under the same account”). Also, I found situations where VisualVM was unable to successfully instrument an application (and I’m not really sure about the reasons). In such cases support for “pre-instrumentation” would be nice.
Also, VisualVM 1.2 introduced the option to use a sampling profiler for even lower overhead.

Personal conclusions:

JProfiler is the best money can buy (it seems that I’m not alone with this opinion), but it is very expensive (especially the “floating” licenses - they are really useful because they allow the product to be installed on an arbitrary number of machines, but only N of them can be used at the same time)
YourKit is mediocre (from the point of view of the capabilities), but it is considerably less expensive (especially the floating licenses)
NetBeans is nice, if you already use NetBeans or you can convince people to use NetBeans. In fact it is comparable with YourKit with regards to the feature set ~~with the two restrictions of local profiling on 1.6~~ and it is free! Alternatively, you can use VisualVM if you are at the start of your profiling journey (for example: you don’t yet have the automated QA environment for performance regression testing).

HTH

PS. And remember: the two most important things for performance optimization are: good data (collected by a profiler in an environment which is as close to the production one as possible - there are a lot of great profilers out there for many programming languages, for example Devel::NYTProof for Perl) and clear goals (along the lines of “lower the latency by 10%” or “increase the throughput by 20%”).

As if you need extra reasons for paranoia

Attila-Mihaly Balazs — Sat, 02 Jan 2010 22:23:00 +0200

As they say: just because you’re paranoid, it doesn’t mean that you’re not being watched! Here are a couple of links which freaked me out recently:

Warily, and with much trepidation - a blogpost about how one deleted his facebook settings because of privacy concerns. I do have a facebook page, but I only use to prevent “identity spoofing” and have almost zero interest in actually using it. Also, on a related note, I’ve read recently that someone was asked a security question by his bank which would indicate that they (whoever “they” might be - the bank, a third party contracted to do it, an information broker, etc) are data-mining facebook (the question was something along the lines of “your sister-in-law’s maiden name” - which BTW I couldn’t answer!)
How to Generate Nice Graphs for Feedburner - nice article about generating graphs (quite nice looking graphs I might add!) with Perl, until you realize that your feedburner statistics is freely available for everyone, just by fetching one URL which has no protection whatsoever! Not that I consider feedburner stats “extremely sensitive”, but it makes you wonder what other data do “helpful” service providers make available about your activities without your knowledge?
How to take down a blogspot blog (like mine for example!)? Just get Google to suspend the associated gmail account! Now, from the writeup it is not entirely clear what happened. Scenario A: someone broke into Mish’s account (by bruteforcing the password / guessing the password reset question / installing a keylogger on his machine / etc) and used it to send spam. Scenario B: someone sent spam spoofing the “from” address (quite easy to do, but most spam filters should detect it). If it was scenario A, we can at least console ourselves with the fact that there is something we can do (like practicing “safe-hex”). If it was scenario B, it means that any two-bit script kiddie can take down any blog… And not every one of us knows Googlers like Mish who can help him sort the situation out quickly.

Now, if you’re still reading and not hiding under a rock scared, here are some links to cheer you up:

Have fun and remember: “The real test of courage is in our daily lives.”

Don’t Yield to pressure?

Attila-Mihaly Balazs — Sat, 02 Jan 2010 20:21:00 +0200

or: does Thread.yield have its place in todays Java programs?

I was profiling a rather old legacy codebase (since the first rule of performance optimization is “profile it” with the close second of “have clear goals in mind” - but that’s an other post) and - after optimizing the first few hotspots, Thread.yield appeared at the top of the most timely methods. I was intrigued, since I didn’t use yield since I wrote “cooperative multitasking” programs for Windows 3.1 in VB 3 (and I’m not a big Python programmer either). So I scoured the ‘net for Information on why/when you should use Thread.yield, but came up with relative few pieces of information:

I found indications that some old Linux kernels behaved poorly on single processor machines if Thread.yield wasn’t used (as in: one thread consuming all the CPU)
There were discussion about using Thread.sleep(0) vs. Thread.yield (apparently there is a difference regarding the treatment of remaining time-quantum by the scheduler)
… this was pretty much it …

So I’ve decided to do some micro-benchmarks. They consisted of a producer and a consumer thread, connected by an unbounded queue (a LinkedBlockingQueue to be more exact) and I measure the number of items produced / consumer in 10 seconds. The first set of measurements were performed on a dual-core machine, while the second set in a VM to simulate a single-CPU system (it’s kind of ironic that one has to perform simulation to evaluate single-core systems). This isn’t meant to be a performance, evaluation, thus all the numbers are normalized to the produced/direct number.

	2 CPUs		1 CPU
Direct	1	1	1	0.06
Thread.yield()	0.12	0.12	0.04	0.04
Thread.sleep(0)	0.1	0.1	0.04	0.04
Priority - 1	0.81	0.81	1	0.05

(My) conclusions:

These days there is no need for “helping” the OS scheduler out. Both of the proposed methods (yield and sleep) reduced the throughput of the system considerably.
Speaking of throughput: make a decision about the (performance) numbers your system should achieve. This includes both throughput and delay. Concrete (and realistic!) numbers. “As good as possible” is not a number! Neither is “better than the current”. Then profile and optimize it until the numbers are achieved and no further.
In the case of a single CPU system there was a big imbalance between the speed of the producer and consumer which Thread.yield (or Thread.sleep) seemed to solve. Consider however, that this “solution” comes at the price of (almost) two orders of magnitude reduction in the throughput. A much better solution would be (in case you are in the rare situation of single CPU - maybe you’re on a VPS, or you have multiple CPUs, but the number of threads far outweigh the number of CPUs) to use a bounder queue. This way the producer gets slowed down (by blocking repeatedly) if it produces faster than the consumer can consume. Then again, you need consider if this is acceptable for your application. Maybe the “overflow” situation is rare and it is more important to handle each element (and there are enough resources for it in the long run) than response speed. You have to know your application and its priorities. There is no way around it.

Finally I would like to leave you with the following short (\~30 min) presentation about performance optimization on the JVM: Making every millisecond count! JVM performance tuning in the real-world.

How to save/restore iptables rules on Ubuntu?

Attila-Mihaly Balazs — Mon, 28 Dec 2009 13:01:00 +0200

This might be an obvious thing to old Linux-heads out there, but it sure caught me off-guard, so there might be some use in spelling it out:

iptables-save and iptables-restore do not actually save/load the iptables rules to/from an external file. You are responsible for redirecting the output of iptables-save to a file and modifying the interface-up scripts such that it is loaded before the given interface comes up.

The Ubuntu documentation tells you how (although, it also was the source of my confusion) - the following commands should be executed as root, so don’t forget to sudo su first:

Save your rules in a file: iptables-save >/etc/iptables.rules
Edit your interfaces file (substitute your own favorite editor here): nano /etc/network/interfaces
Add a pre-up command to restore the saved rule. The fully configured file should look similar to this (the bold line is the one added):
```
auto eth0
iface eth0 inet dhcp
  pre-up iptables-restore < /etc/iptables.rules
```

HTH. And remember - security is a process / mindset, not a state. Always test the configuration changes you’ve done, don’t just assume that everything went ok because you didn’t receive error messages.

How eco-friendly is a BMW?

Attila-Mihaly Balazs — Sat, 26 Dec 2009 22:06:00 +0200

[![I WANT A BMW ISETTA!](http://farm1.static.flickr.com/77/218926578_b24658ed93.jpg)](http://www.flickr.com/photos/maazbot/218926578/ “I WANT A BMW ISETTA! by maazbot, on Flickr”)

The short answer is: I don’t know :-)

While I was watching National Geographic, I caught a glimpse of the BMW “Efficient Dynamic” advertisement campaign. The claims made by this campaign were quite extraordinary and - being the cynic that I am - I thought: hang on, this sounds too good to be true. The claims as I recall were:

BMW reduced fuel consumption by 16%
This reduction is more than twice the reduction achieved by the next premium segment competitor
This reduction is more than twice the average reduction obtained by the industry

Being an aspiring skeptic I decided to look into these claims, but being the lazy ass that I am, quickly gave up after making a mental list of what would be involved (finding out what they mean by “premium segment” and who their competitor were, finding a reliable source of data, etc). So, instead, I turned to math to see if all these claims can be true at once. So, in math-talk we have the following data:

BMW = 16
Lets suppose that we have three competitors A, B and C with A being the closes to BMW
A, B and C are in the interval [0, 100]
BMW >= 2*A
BMW >= 2 * AVG(BMW, A, B, C)

Then I turned to the OpenOffice Solver which promptly came up with an answer: A=8, B=0, C=0. Starting from this I came up with a more plausible-looking solution: A=7, B=5, C=5.

What does this mean? That - mathematically speaking - the claims made might be true. As always - trust, but verify. These simple mathematical tools are available to everyone and can be used to unmask the more extreme false claims (of course, just because a claim is mathematically possible, it doesn’t make it necessarily true). Go search for information. You should find it - since it wants to be free!

Picture taken from maazbot’s photostream with permission.

Recouping your data from a hung program

Attila-Mihaly Balazs — Sat, 26 Dec 2009 21:15:00 +0200

Scenario: you are typing away in your blog editor on Ubuntu doing a (somewhat) Flash-heavy post. You make the mistake of hitting “Preview” and the blogging software hangs. How can you get your post out?

Find the PID of your blogging software
Coredump it (gcore [PID] - this will create a file called core.[PID] in the current directory) - sidenote: interestingly, coredumping doesn’t actually kill the application - this makes me wonder about thread safety… What guarantees does gcore make about the consistency of the dumped state? Probably none… This isn’t important in this case, since the program is hung for good.
Use a hex editor (GHex for example) and search for a part of the blogpost. You will probably find it multiple times, but you can easily identify one occurrence which has a complete copy.
Copy the blogpost from the hexeditor
Profit!

Hope this saves somebody from retyping their text!

PS. This can be applied to other programs too where the storage format is “human readable” (like text editors - as opposed to spreadsheet editors). An other trick you might try is to search for the string as Unicode (since more international-aware programs might store it as that). While GHex doesn’t support this directly, you can manually insert the 00 bytes between the Latin characters. An other option would be to run strings on the coredump file with different --encoding options.

Congratulation to AV-Comparatives!

Attila-Mihaly Balazs — Fri, 25 Dec 2009 20:06:00 +0200

AV-Comparatives is an independent, well-known and well respected testing organization in the AV/Anti-Malware field. They recently published two reports and one meta-report:

Go read them if you have questions like “which product is the best for me?”. Thank you Andreas for providing a great and impartial service.

PS. One surprising thing for me was the high detection rates in the dynamic test - upward of 90%. This indicates that either I’m too much of a cynic or that their crawler system still has room to improve - I would expect AV products to be around 60-70% effective against new threats.

Don’t listen alone!

Attila-Mihaly Balazs — Fri, 25 Dec 2009 19:47:00 +0200

Do you like Linux? Do you listen to podcasts? If you’ve answered yes to both of those questions, you should know what LUG Radio is (if not, do a quick checking - I promise you that it will be worth it!).

The bad news? They stopped it in 2008. The good news? A documentary titled “Don’t listen alone!” - a great title if I may say so - about it just came out! So watch it below (sorry for splitting it up into 10 minute segments, but YouTube limits you to this):

http://www.youtube.com/p/F4B2E3619B9D2E49&hl=en\_US&fs=1

Or go over to Jono’s site and watch it from blip.tv (my problem with blip.tv is that their delivery method seems to be much less bandwidth friendly - I’ve got constant “buffering” even on connections where YouTube HQ clips play fine) or download it from archive.org. You can also read up on how the documentary was created (on Linux!) here.

Finally, if you still miss their voices (as I do), head over to ShotOfJaq or to FLOSS weekly and you will be pleasantly surprised!

PS. Offtopic rant: I’m all for open formats and such, but when - after days of searching! - I can’t find a tool which supports the OGV container (or the Theora codec for that matter) properly, I’m tempted to give up on them! On the AVI/XVID/h264 side there is Avidemux for example… Finally I had to re-encode the whole video into AVI/XVID just be able to chomp it into 10 minute segments.

Schneier videos

Attila-Mihaly Balazs — Mon, 21 Dec 2009 17:36:00 +0200

Bruce Schneier is always fun, and together with Markus Ranum he is extra fun (sidenote: although it is title “face-off”, they agree more than they disagree):

And here are some Schneier only videos (the first video has some audio problems in the first 3 minutes, but it gets better afterwards):

http://vimeo.com/moogaloop.swf?clip\_id=8053634&server=vimeo.com&show\_title=1&show\_byline=1&show\_portrait=0&color=&fullscreen=1 [Open Rights Group: Bruce Schneier Security Talk](http://vimeo.com/8053634) from [Open Rights Group](http://vimeo.com/user1287766) on [Vimeo](http://vimeo.com).

http://vimeo.com/moogaloop.swf?clip\_id=8062617&server=vimeo.com&show\_title=1&show\_byline=1&show\_portrait=0&color=&fullscreen=1 [Open Rights Group: Bruce Schneier Security Talk (Q&A;)](http://vimeo.com/8062617) from [Open Rights Group](http://vimeo.com/user1287766) on [Vimeo](http://vimeo.com).

New challenges

Attila-Mihaly Balazs — Fri, 18 Dec 2009 18:15:00 +0200

After missing the announcement for the second part of the Network Forensics Puzzle (yes, I’m subscribed the feed now!) I would like to regain your trust by bringing two other contests to your attention:

Miracle on Thirty-Hack Street from ethicalhacker.net
the sevenfour challenges (a “keygen-me” type of challenge)

Bonus content:

t2’09 challenge solutions (via the Reverse Engineering Reddit)
Hacking Challenges: Have Fun Improving Your Skills! [PDF] – presentation from the RaDaJo guys

Have fun!

Picture taken from ChrisDag’s photostream with permission.

A game of Chinese whispers

Attila-Mihaly Balazs — Fri, 18 Dec 2009 17:23:00 +0200

Yet an other example of real-life Chinese whispers in the security journalism:

A Hungarian online news site published an article titled “Hackers tried to steal user data from Amazon” (here is a somewhat usable automatic translation for the non-Hungarian speakers). I assume that the information went like this:

What happened –> What the security company has written up about it –> What the “journalist” understood –> What s/he actually wrote.

What actually happened is that an Amazon EC2 rented to a third party was being used as a C&C; server for a botnet. No Amazon user data compromise here, move along (also, this isn’t a new phenomenon at all).

To top it off, the article talks about the security issues involved in cloud computing. Surely they are paid by buzzwords / paragraph :-p.

As if you needed further proof that a large percentage of the news out there is false, even when there is no intent to “spin” it. Newer attribute to malice what can be explained by stupidity I suppose…

Picture taken from bignoseduglyguy’s photostream with permission.

Twitter hacked

Attila-Mihaly Balazs — Fri, 18 Dec 2009 13:16:00 +0200

![DiggThis](http://digg.com/img/diggThis.png)

It had to happen, didn’t it? I’ve fired up Pidgin with the microblog-purple plugin, only to get an “invalid certificate” error for twitter. I’ve quickly became nervous, since a quick digging indicated that I was getting the wrong IP address for the domain twitter.com.

My first thought was: “I’ve been compromised”. After quickly verifying my hosts file and my DNS entry, all seemed fine on the surface. My second thought was: “my DNS server was compromised”, so I’ve done the same lookup using OpenDNS and the new Google DNS, both coming up with different (but wrong) answers. Finally I’ve checked out a couple of other HTTPS sites and they seemed fine. So I took a deep breath and (putting my faith in NoScript and RequestPolicy) visited twitter.com to find the following page:

Quick analysis:

This seems to be a “good old” defacement
A very likely scenario is that they somehow compromised the DNS registrar account (phising, dumb password reset, etc) and changed it to point to an other IP.
Currently I’m seeing a couple of different IPs out there for the twitter.com domain:
- My DNS server (and OpenDNS) returns 66.147.242.88
- Google DNS returns 74.217.128.160
The correct address seems to be 168.143.171.84, so if you put the following line in your host file, thing should start working again (you might need to do an ipconfig /flushdns if you’re on Windows):
```
168.143.171.84 twitter.com
```
The above is a hackish solution, and I would recommend using it only in life-and-death situations :-p. It is the best to let Twitter handle the incident and make sure that everything is cleaned up.
It is unclear when exactly the defacement happened, but it must have been in the last 10 hours or so. It might have been specifically targeted so that it is late in the day in the USA so that the reaction is delayed.
~~According to Google Translate (Babelfish doesn’t know Arabic unfortunately) the text below the picture says:~~

Ok, so I’m a big ignorant idiot. The official language of Iran is Persian (also known as Farsi or Parsi), not Arabic. Thank you to Anonymous for pointing it out. According to this article the text in the picture says:

This site has been hacked by the Iranian Cyber Army (on the flag)

and

The USA thinks they control and manage internet access, but they don’t. We control and manage the internet with our power, so do not try to incite the Iranian people (under the picture)

Some people also seem to have screenshots with English texts on them.
The rogue server doesn’t seem to respond to any Twitter API requests, so it doesn’t seem to be that they were going after usernames and passwords (which they very well might have done, considering the number of users who click trough SSL certificate warnings), but just to be on the safe side, change your password and don’t use the same password on all the sites!

Update: As of now all seems to be back to normal and all the DNS servers return the correct IP address. I’m waiting for an explanation in Twitter (mostly because I’m interested in how it happened :-)).

Update: Twitter acknowledges the hack on their blog and say that they will provide more information as it becomes available (however they erroneously affirm that the API were working correctly – they weren’t, since they used the same DNS record to contact Twitter – in fact this is how I’ve became aware of the hack).

Bonus: what sources can you use to investigate such incidents?

First of all, be suspicious of SSL certificate errors! I know that they (sadly) are quite common these days, but be vigilant!
Check that the problem is not at your end. Check that you have the correct DNS server (there are a couple of malware families out there which set a custom DNS server for the machine to control the users browsing destinations). Check that the given hostname is not present in your hosts file (again, there are a couple of malware families using this method to misdirect users)
Check what the IP address should be, by using domaintools for example (and looking at the server stats page)

Try looking up the DNS name using several DNS servers (this might not work if your network filters DNS queries):

# nslookup
> set type=ANY
> twitter.com
...
> server 8.8.8.8
> twitter.com
...
> server 208.67.222.222
> twitter.com
...

An other option is to use the vURL service to fetch the suspicious webpage from different location and compare the results with what you are seeing.

Using these methods you can quickly ascertain with pretty good accuracy where the fault lies and take appropriate action. Have a safe holiday everybody!

Update:

Read about the subject on the TrendMicro Countermeasures Blog.
Some more links to information and the source of the defaced webpage at Hacker News.
SANS posted about in issue in the diary.
I’ve update the translations, thanks to Anonymous
Twitter posted an update about the issue. It doesn’t many more details, it does however give a timeframe for the problem: between 21:46 and 23:00 PST . There are some rumors out there that somehow (phising?) the correct password to the DNS management interface was obtained and it was used to modify the records. Twitter still has the original blogpost up saying that API’s were not affected, but this is not true! If you’ve used a third party Twitter client and you’ve clicked trough the certificate warning (or maybe it doesn’t use TLS at all), your password might have been compromised. Currently there is no evidence that the rogue server was logging passwords, but until the time some forensics is done on it, there is no sure way to tell if this was the case (since it is trivial to configure a webserver such that it responds with a 404 error, while still logging the details of the request).
Arbor Networks posted a related article.
Sucuri has also posted about the issue. They have a nice little network monitoring / alerting system. You can also use them as a third-party information source.
ISS X-Force (part of IBM) has also a nice writeup about the incident.
Brian Krebs has an informative writeup on the SecurityFix blog about the issue which quotes Dyn’s (the host for the Twitter DNS) CTO as saying: “Someone logged in who purported to be a legitimate user of their [DNS] platform account and started making changes”, further strengthening the probability that a Twitter employee’s email account was broken into via some mechanism.
There is also a lot of confusion out there, as it always is the case with (security) news. I’ve heard someone saying that “why did the DNS host allow the redirection of Twitter to a host in Iran?” - just to clarify: even though the hack was claimed by the “Iranian Cyber Army” (which might not mean anything! it could be your nerdy neighbor), the server it was redirected to was in the US.

Picture taken from pugetsoundphotowalks’ photostream with permission.

Discount Codes UK review

Attila-Mihaly Balazs — Thu, 17 Dec 2009 13:41:00 +0200

These days most online shops offer the ability to use discount codes at checkout and get a price reduction anywhere from 5% to 50%. These codes are announced in various media (like podcast or blogs), but even if you don’t follow the particular program, it is rather easy to find them with a search engine.

Given these premises, sites like Discoount Codes UK are welcome. I didn’t use their services personally, but it is well organized with direct links from the discount code to the store where you can use it and good reputation on sites which track such things (MyWOT, Norton SafeWeb and SiteAdvisor). For added safety I would recommend entering the addresses of the shops manually, rather than using the provided links. So there you have it: discount codes at your finger tips with the possibility to get a couple of percents off. And even if some don’t work, you have nothing to loose by trying to use them. Again a word of caution, especially around holiday shopping: be cautious and either use big-brand shops (like Amazon) or thoroughly check out the given shop (using a search engine and searching for phrases like “[shopname] complaints”, “[shopname] problems”, “[shopname] fraud”, etc). Better safe than sorry!

Full disclosure: this is a paid review from ReviewMe. Under the terms of the understanding I was not obligated to skew my viewpoint in any way (ie. only post positive facts).

I’m the spam killa’

Attila-Mihaly Balazs — Tue, 24 Nov 2009 13:05:00 +0200

I’m happy to announce that I’m one of two “spam killers” on the Software Engineering radio website. Spam was starting to run rampant on their site, so they asked for help and I responded. It is so simple to donate your time to a worthy cause. You to can do it, it takes just a couple of minutes per day!

PS: If you are interested in software development / design, this is definitely a podcast you should give a listen.

Picture taken from Manuel_Marin’s photostream with permission.

Screenshot forensics

Attila-Mihaly Balazs — Tue, 24 Nov 2009 12:48:00 +0200

One of the interesting thing I like to do when reading (security) blog posts, is to try to deduce details about the machine setup used. You can find some very interesting tidbits of information, like Sunbelt using Symantec AV on some of their machines.

A couple of current examples:

a CA researcher uses Office 2007 and Google Chrome
a Sophos researcher seems to prefer Ubuntu, or at least a Gnome based desktop

If you want to avoid exposing such details, try the following:

Crop the screenshot as much as possible. This has other advantages as well (smaller image size which leads to quicker display for example)
Remember that identification can be done in any number of ways:

NET

If you are really paranoid, you might want to consider taking the screenshot on an entirely different OS (Haiku for example :-).

Got fun “screenshot archeology” findings? Share them in the comments!

Picture taken from DeusXFlorida’s photostream with permission.

Plugging a good friend of mine (not in a sexual way! :-P)

Attila-Mihaly Balazs — Mon, 23 Nov 2009 17:45:00 +0200

A talented photographer with a lot of beautiful images. Check them out below or on his flickr stream. Go OPE!

http://www.flickr.com/apps/slideshow/show.swf?v=71649

Today’s fudbuster

Attila-Mihaly Balazs — Mon, 23 Nov 2009 17:24:00 +0200

We begin today’s FUD-buster with – applause please – cyberterorism via an “article”: Cyberterrorism: A look into the future. The article talks about Estonia (which is the poster-child for “cyber” incidents these days) and says the following thing (amongst others equally high-quality content) – emphasis added:

“The three-week cyberattack on Estonia threatened to black out the country’s digital infrastructure, infiltrating the websites of the nation’s banks and political institutions”

The article cites as source (hey, at least they cite sources) an equally “well researched” piece from the Telegraph.co.uk which says almost the same thing. Now I seem to remember that the Estonia incident was just a large scale DDoS attack, so I’ve looked around for more reliable sources, like this article on Dark Reading Authoritatively, Who Was Behind The Estonian Attacks? by Gadi Evron (or see this other article). This confirms what I was remembering: it was a large scale DDoS attack with some minor defacements, but in no way were they “infiltrating the websites”.

The second (unrelated, other than the fact that it is an overstatement) quote comes from the Kaspersky blog, where we can read that:

“a vast amount of pirate software nowadays contains trojans, both for the PC and Mac”

This depends very much on your interpretation of “vast amount” (as me how I know :-P). Of the actual pirated software shared in limited networks like college campuses, very little is infected. What are extremely likely to be malicious are the crack / keygen websites. Either they contain exploits directly or they bundle malware with the downloads. An other sneaky way, seen on P2P networks like Gnutella or eDonkey, is to run bots which respond to any search with an executable that contains the keywords in the name and is – of course – malicious. So, depending on your interpretation of “vast amount”, this doesn’t hold up.

The conclusion, as always: do your own research!

Picture taken from cooljinny’s photostream with permission.

ActivTrack review

Attila-Mihaly Balazs — Mon, 23 Nov 2009 16:06:00 +0200

ActivTrak is an activity tracking and employee monitoring software. It currently supports the 32 bit versions of Windows 2000, XP and Vista with support for 64 “coming soon” (no word on support for Windows 7 as of yet). The features are the basic ones one would expect from such a product:

direct deployment from the management console (however this can become tedious for a large number of computers)
tracking active programs / windows and URLs (in case of browsers)
taking periodic screenshots
basic reporting about the data

One nice thing is the fact that it employs a “reverse connection” (ie. the server opens up a port and the clients connect to it). This has the benefit of requiring less configuration on the clients and making them more secure (also, the server configuration part is done during install automatically). While trying out, you can run both the viewer and the agent on the same machine (it will report it as “no running”, but the data will still be available). You can watch multiple workstations at once by tiling the screenshot windows and setting them to auto-refresh.

Two shortcomings of the program are the fact that (from what I understand) the server needs to be running continuously for data collection (then again, it might be just a misunderstanding on my part, but this was the impression I got). The second shortcoming (maybe its not a shortcoming, but definitely something to be aware of) is the fact that you have very limited interaction with the surveyed computers: no controlling the mouse / keyboard / file-transfer. You can send messages and chat with the user. This means that the product can’t be used directly in a “support” type environment.

A final word of caution: consult with a lawyer before deploying such a solution (it might be illegal depending on the circumstances!). Also, consider the impact on the morale. If you have staff which needs this level of constant supervision, you might be better looking for new employees.

Full disclosure: this is a paid review from ReviewMe. Under the terms of the understanding I was not obligated to skew my viewpoint in any way (ie. only post positive facts).

Small Business VoIP review

Attila-Mihaly Balazs — Mon, 23 Nov 2009 15:11:00 +0200

I have posted reviews trough ReviewMe for VoIP products before, but here is an other other one: Vocalocity is a provider specialized on small business voip. They’ve been in business since 2005 and all the reviews about them which I could find were glowing (one might suspect foul play given all the good reviews, but digging deeper some of them mention problems and specify that the support was very good and helped them trough the hiccups). Of course if you had a negative experience with them, please share it in the comments.

An other positive aspect is that their pricing plan is prominently featured on their webpage, so you should have no problem finding it. They also pride themselves with not being resellers (“owning their own technology”) and have a nice office building:

[View Larger Map](http://maps.google.com/maps?hl=en&ei=7pgWStaEGM_Jtgfy-PH0DA&q=1375+Peachtree+St+NE,+Suite+175,+Atlanta,+GA+30309&ie=UTF8&hq=&hnear=1375+Peachtree+St+NE,+Atlanta,+Fulton,+Georgia+30309&ll=33.792416,-84.386126&spn=0.012732,0.027874&t=h&z=16&layer=c&cbll=33.792222,-84.386175&panoid=gCQM_44IEoddVHj5r1iFzA&cbp=12,65.36,,0,5.07&source=embed) What else is there to say about them? If you are looking for a way to reduce the complexity of your phone network, take a look at them (of course, you have to consider other aspects of your business – like what level of guarantee you need that others won’t access the voicemail – the assurance you *can* provide in-house is always greater, at a higher cost of course). *Full disclosure: this is a paid review from* [*ReviewMe*](http://www.reviewme.com/)*. Under the terms of the understanding I was not obligated to skew my viewpoint in any way (ie. only post positive facts).*

To my dear wife

Attila-Mihaly Balazs — Sat, 21 Nov 2009 09:54:00 +0200

http://listen.grooveshark.com/songWidget.swf

If you are viewing this from the RSS feed: please visit the blog to see the embed. Many RSS readers filter out embed codes.

Calls to action

Attila-Mihaly Balazs — Tue, 17 Nov 2009 18:41:00 +0200

With the motto “better late than never” here are some calls to action:

Vote for your favorite podcast on the Podcast Awards website. Votes are open until November the 30th and you can vote once per day (after you vote, you can an email with a link, which you must click on to validate your vote – this is to reduce the number of “fake” votes). If you are unsure for which podcast to vote, here are some suggestions: in the “Best Video Podcast” category I would recommend Buzz out loud – it is a very good (informative and fun) daily tech-news podcast. In the “Business” category I would recommend Career Tools- it (together with its sister podcast Manager Tools) is a great resource. In the Technology category I would recommend FLOSS Weekly – it is a superb podcast for all people interested in free / libre / open-source software. And it would be a great gift for them for the 100th episode which is quickly approaching. And besides – TWIT already won a couple of times :-). So go ahead my ~~minions~~ readers, fly like the wind and vote!
And here is a second poll related to Perl IDE’s: What other technologies, languages, templating systems are you using besides Perl?

After you have done your deed :-D, you can relax with two fun flash games: Little Wheel, a fun old-school point-and-click adventure game with very nice artwork (including an interesting soundtrack). Or play nine-balls. Let the lightning be with you!

[![Little Wheel](http://www.fastgames.com/images/littlewheel.jpg) Little Wheel](http://www.fastgames.com/littlewheel.html) [![Billiard Blitz 3 - Nine Ball](http://www.fastgames.com/images/billiardblitz3-nineball.jpg) Billiard Blitz 3 - Nine Ball](http://www.fastgames.com/billiardblitz3-nineball.html)

Surprising numbers

Attila-Mihaly Balazs — Tue, 17 Nov 2009 18:05:00 +0200

I was reading the latest FudSec piece (Generating a False Sense of Insecurity) where I found the following statement (emphasis added):

Facebook now has 300 million users. Let’s assume that each user has at least one piece of user-generated content on their Facebook page cause, well, it’s a very user-content driven site. That means that of the 300 million home pages on Facebook that 95% (285 million) has either a malicious link or other insecure content. Conversely that means that 5% (15 million) are clean, uninfected, safe pages.

The average Facebook user has 120 friends or 281 friends, depending on which news article you might be reading. Let’s just assume for mathematical purposes that the number is somewhere in the middle, at about 200 friends per user. Let’s pretend, too, that you visit every friend’s page in a single day. Because it’s your day off, of course, you wouldn’t actually do that at work.

The mathematical likelihood that one of your 200 friends is one of the 95% that is infected is infinitesimal.

This statement seemed a little off. After all, we are selecting 200 pages out of 300 million where 275 million are infected. The chance to get to an infected / malicious page can’t be that low, right? Wrong! The problem as stated is known in mathematics (probability theory to be more precise) as the “drawing without replacement” and apparently the scientific name is hypergeometric distribution. Long story short, Wikipedia pointed me to a calculator which says that – given the parameters quoted above – you have a 99.9999608980365% chance that all of your friends will be clean / non-malicious! Talk about counter-intuitive!

Conclusion? First of all, trust but verify. If you hear something which sounds “off”, try to verify the information from multiple sources. Then again, our brains don’t seem to be wired to evaluate probabilities “heuristically”, so one should always sit down and work out the exact math (there are a lot of free tools on the Internet which can help you) before making important decisions.

Picture taken from EraPhernalia Vintage’s photostream with permission.

Web Hosting Site Review Review :-p

Attila-Mihaly Balazs — Tue, 17 Nov 2009 12:03:00 +0200

WebHostingChoice pretends to be a hosting review site (it contains categories like “best uk web hosting”), however it only seems to be a placeholder for a couple of affiliate links to a limited number of hosts. Their WOT (web of trusts) rating isn’t so great either. While WOT has its limits (mainly because of its “crowdsourced” nature), when it has several negative ratings, it can be a good indicator that there are problems with the given site.

So how to find a good webhost? First of all, you should realize that (usually) you get what you pay for (ie. “free” webhosts are rarely free). I would recommend going with a “big brand” company like GoDaddy or Rackspace. You can fairly easily find coupon codes for them (just listen to some technical podcasts) which can get you a considerable percentage (like 20%) off. An other company which seems very good is firehost. While they are a little pricy (especially compared to the other two companies), they consider security an explicit priority, which is very important these days IMHO – if they are willing to take on people with large targets on their back like Kevin Mitnick, they should be able to protect your business too.

Full disclosure: this is a paid review from ReviewMe. Under the terms of the understanding I was not obligated to skew my viewpoint in any way (ie. only post positive facts).

The leaked Microsoft COFEE product

Attila-Mihaly Balazs — Mon, 09 Nov 2009 14:09:00 +0200

So, the Microsoft COFEE (Computer Online Forensic Evidence Extractor) tool was leaked. I took a quick look at it, and – as expected – there is nothing “magical”, “secret” or “backdoorish” about it (even though I love the picture which comes with the Gizmodo article, the text itself is complete and utter BS – COFEE isn’t a tool “that helps law enforcement grab data from password protected or encrypted sources” as the article claims).

So what is Microsoft COFEE?

it is a collection of information gathering tools which are either built into Windows (ie. net, arp, ipconfig) or can be freely downloaded from the Microsoft website (ie. pslist)
it contains a simple case-management software which helps users prepare a USB stick that need to be inserted in the target computer and manage the collected information
the software on the USB stick is executed either using the autorun mechanism or by manually launching it. There is no built-in functionality to bypass passwords or other protection mechanisms
It also contains a detailed analysis of the registry / filesystem fingerprint of each tool (this is important if the other party argues that running the tool caused modifications on the system which are pertinent to the case)

Conclusion: there is no magical pixie dust here, move along! (in fact, it is quite similar with the winenum Metasploit script).

PS/Update: regarding the “defense” against these tools: first of all, they all seem to be user-mode tools. This means that they probably have limited capability of detecting kernel-mode rootkits. Also - from what I’ve seen - they are all public tools, so there is a good chance that there exists malware out there there which “defends” itself against these software. Again, no magic.

Now before you conclude that this is utterly useless - if I were a IT forensicator :-p, I would prefer having this data compared to no data at all. It will give you some basic idea of the system (or the network for that matter if ran on every PC) which may enable you to come back with a very precise target in mind.

Picture taken from raddaqii’s photostream with permission.

What VirusTotal is not

Attila-Mihaly Balazs — Mon, 09 Nov 2009 13:34:00 +0200

Since its inception VirusTotal has been used by people to compare different AV products (just in case you don’t know: VirusTotal is great free service which scans the uploaded file with 40 AV engines currently and reports back the results). The AV industry has objected to this practice because of a couple of reasons, some more valid than others IMHO.

Today however I want to talk about the practice of saying “(only) X% of AV detect this” and then giving a VirusTotal link. Two recent examples: here and here (to be clear: I don’t have anything against the particular blogs / companies / authors – there are many more examples of this practice, these are just two recent ones which came to my attention).

Why is this percentage meaningless and serves only to perpetuate FUD?

As I first argument I could mention all the discussion about AV engine configuration (this is frequently raised in discussion regarding the detection discussion, so I won’t dissect it further). A very thoroughly discussed argument is also that VT results represent a “point in time” rather than “now” (ie. detections since the scanning might have changed).
The second argument would be: VirusTotal goes for quantity not necessarily quality. Ie. the fact that a given engine is included in the list of engines used by VirusTotal isn’t a statement about the engine resource use, detection rate or false positive rate. Again, this doesn’t mean that the engines used are of low quality, it just means that VirusTotal isn’t in the AV engine testing business. It doesn’t say anything about the market share of the product either.
This means that the affirmation “X% of the engines detect a given file on VT” isn’t equivalent with the affirmation “X% of the users using AV are protected” or “AV software is X% effective”. However these are the thoughts which appear (by association) in a readers mind when seeing the initial affirmation.
Furthermore, some engines appear in multiple products (for example GData integrates BitDefender – amongst others) while other engines appear “split” (for example the McAfee desktop product contains both the “classical” and “cloud” engine, however on VT they appear as two separate entries “McAfee” and “McAfee+Artemis” respectively). If these relations are not considered (and I’m almost sure that they aren’t – given that these relations are not always publicly documented and they can change over time), the results come out skewed.

Conclusion: please never, ever take the VT result page and copy-paste the percentage from it! Do provide permalinks to the result pages and you can even make some sensible general statements (like “most of the major AV vendors detect this threat” or “this threat is not well detected by the smaller, Asian AV companies, but given its reliance on the English language for social engineering, it might not be such a big threat”). However, giving percentage wreaks of FUD and smells of negative propaganda (do we really want to be at each-others throat, analyzing which vendor doesn’t detect what? – there would be no winners in such a discussion). Lets concentrate on giving sensible security advice to users instead.

Picture taken from Peter Kaminski’s photostream with permission.

Grooveshark VIP member

Attila-Mihaly Balazs — Mon, 02 Nov 2009 16:22:00 +0200

I’ve written about Grooveshark in the past, however I want to mention them again for a couple of reasons:

First of all, they introduced a new user interface, which works great. More than that, you can now seek in the songs! This means that Grooveshark directly addresses three out of the five methods of music use which I’ve enumerated in my original post. There are some small quirks (I don’t really like the popup-type controls, where you first have to hover over it for the useful part to appear), but those are just a matter of personal taste. They’ve also made it available as a desktop application via Adobe Air (currently available only for VIP subscribers).

Which brings me nicely to my second point: I’ve subscribed to their VIP services. I thought that I’ve been using them for a month now and I’m satisfied, so I should give something back aka. “Vote with my money”. So, as of today, I’m a Grooveshark subscriber. A couple of things I didn’t like about the subscription process: there is an additional tax of 15% to the advertised 3 USD monthly price. Also, the subscription payment is set as recurring by default. You can deactivate it later, but even so, it made me feel a little uneasy. Still, I decided to give them some of my money. Hopefully I won’t regret it.

As of now, I can only recommend Grooveshark to everybody! If something happens, I will update this blogpost.

PS. I’ve also removed the last.fm widget from my blog. Currently Grooveshark seems to be a much better deal than last.fm for approximately the same amount of money.

Disclaimer: I don’t receive anything from Grooveshark, I’m just a happy subscriber.

How to generate a stackdump with GDB

Attila-Mihaly Balazs — Fri, 30 Oct 2009 17:29:00 +0200

I’m not a big GDB guy, but Google always helps:

Create a textfile with the following content:

set height 0
thread apply all bt
detach
quit

Run the following command:
```
gdb $EXE -pid $PID -command $TEXTFILE > $OUTPUTFILE
```
where:
- \$EXE is the path to the executable
- \$PID is the PID it is running under
- \$TEXTFILE is the file where your’ve saved the previous commands
- \$OUTPUTFILE is the file where you would like your stackdump to be saved.

The cool little crawling logo was taken from HiR, head over there for an explanation.

The importance of false positives

Attila-Mihaly Balazs — Fri, 30 Oct 2009 16:11:00 +0200

An interesting paper was bought to my attention recently by this blog post: The Base Rate Fallacy and its implications for the difficulty of Intrusion Detection. The central question of this paper is: if we have a flow of N packets per day and our network IDS has a false-positive rate of X, what is the probability that we are experiencing a real attack, given that the IDS says that we are? The paper uses Bayes’ theorem (of which you can find a nice explanation here) to put some numbers in and to get horrifying results (many false alerts), and to conclude that such a rate of FPs seriously undermines the credibility of the system.

The issue of false positives is also a concern in the anti-malware industry. And while I rant quite a bit about the AV industry, you have to give this one to them: the number of false positives is really low. For example, in the AV-Comparatives test 20 false positives is considered many~~, even though the collection is over 1 500 000 samples (so the acceptable FP rate is below 0.0015%!)~~. Update: David Harley was kind enough to correct me, because I was comparing apples (the number of malware samples) to oranges (the number of clean files falsely detected). So here is an updated calculation: the Bit9 Global File Registry has more than 6 billion files indexed (they index clean files). Consider whatever percent from that which is used by AV-Comparatives for FP testing (as David correctly pointed out, the cleanset size of AV-Comparatives is not public information – although I would be surprised if it was less than 1 TB). Some back-of-the-napkin calculations: lets say that AV-Comparatives has only one tenth of one percent of the 6 billion files, which would result in 600 000 files. Even so, 20 files out of 600 000 is just 0.003%.

Now there were (and will be) a couple of big f***-ups by different companies (like detecting files from Windows), but still, consumers have a very good reason to trust them. Compare this with more “chatty” solutions like software firewalls or – why not – the UAC. Any good security solution needs to have at least this level of FPs and much better detection. AV companies with low FP rates – we salute you!

http://www.youtube-nocookie.com/v/xMUgmU_Hsjc&hl=en&fs=1&color1=0x234900&color2=0x4e9e00#t=1ms50

PS. There might be an argument to be made that different false-positives should be weighted differently (for example depending on the popularity of the file) to emphasize the big problems (when out-of-control heuristics start detecting Windows components for example). That is a valid argument which can be analyzed, but the fact remains that FP rates of AV solutions, is very low!

Picture taken from wadem’s photostream with permission.

Fun videos

Attila-Mihaly Balazs — Thu, 29 Oct 2009 15:08:00 +0200

Got the links from a friend:

http://www.youtube-nocookie.com/v/7H0K1k54t6A&hl=en&fs=1&color1=0x234900&color2=0x4e9e00

http://www.youtube-nocookie.com/v/GepKz9qsQQ4&hl=en&fs=1&color1=0x234900&color2=0x4e9e00

Why network neutrality is a big deal

Attila-Mihaly Balazs — Thu, 29 Oct 2009 09:54:00 +0200

Reposted from the packetlife blog. We already pay for the bandwidth. The content providers already pay for the bandwidth. Anyone claiming anything different is either very misinformed or is straight out lying!

Bohemian Bankruptcy

Attila-Mihaly Balazs — Wed, 28 Oct 2009 17:43:00 +0200

http://www.youtube-nocookie.com/v/w5EFEQ9aY6o&hl=en&fs=1&color1=0x234900&color2=0x4e9e00 Via [Naked Capitalism](http://www.nakedcapitalism.com/2009/10/bohemian-bankruptcy.html)

RequestPolicy Firefox Plugin – the ultimate NoScript

Attila-Mihaly Balazs — Wed, 28 Oct 2009 16:54:00 +0200

I recently found out about the following Firefox plugin/addon: RequestPolicy (via this blogpost) – see also the Firefox addon page. Its function is to whitelist all kinds of cross-domain requests, including scripts, style-sheets, images, objects (Flash, Java, Silverlight), etc. Anything in a webpage hosted on the domain A can reference other content from domain A, but if it references content from other domains, it must be present in the RequestPolicy whitelist. There are three types on entries which can be added to the whitelist:

source (ie. pages on domain S can reference anything)
destination (ie. anything can reference domain D)
source-to-destination (ie. pages on domain S can reference resources on domain D)

There are still some glitches to work out, but all in all it is a good tool for the security conscious. So is it worth it? It depends. If you are not a power-user who has some knowledge HTML (ie. how CSS, HTML, JS and plugin objects fit together to form the page), I would recommend against it (because you will have the experience of webpages “not working for no good reason”). It takes some initial training (just like NoScript), but after that it is pretty invisible (even though not as invisible as NoScript, because it blocks images / style-sheets).

Does it make you more secure? Yes, but just in the “you don’t have to outrun the bear”: once the attacker has enough control to insert a linked resource (script, iframe, etc) in a page, s/he almost certainly has enough control to insert the attack script directly in the page, rather than linking to it. The current practice of linking to a centralized place is mostly because the attackers want to have centralized control (for example to add new exploits) and statistics. Would such a whitelisting solution to become widely used, they could switch with very little effort to the “insert everything into the page” model. Still, such a solution shouldn’t be underestimated, since it gives an almost perfect protection under the current conditions.

Update: If leaving digital trails is something you like to avoid, take into consideration that the fact that a given site is present in the whitelist of addons such as NoScript or RequestPolicy can be considered proof that you’ve visited the given site (unless it is on the default list of the respective addon). Just something to consider from a privacy standpoint. Life is a series of compromises and everyone has to decide for herself how to make them.

Picture taken from Luke Hoagland’s photostream with permission.

help build the mozilla developer network

Attila-Mihaly Balazs — Wed, 28 Oct 2009 16:24:00 +0200

After asking you if you use Perl, now I’m asking you to help build the Mozilla Developer Network (MDC). They are running a survey to get to know their audience better. Please take it if you use the MDC and have a couple of minutes of free time.

PS. You can read some preliminary results from the Perl IDE poll here.

Taking apart the Dell Inspiron 9400

Attila-Mihaly Balazs — Mon, 26 Oct 2009 11:12:00 +0200

A word of caution: taking apart your laptop will void your warranty. Do this operation at your own risk. If you are not comfortable doing this operation, I would recommend against it. Disassembling a laptop is harder than taking apart a desktop computer (mostly because of the confined space), so you shouldn’t do it if you didn’t “look into” atleast couple of desktops already!

You can see a high resolution of the images below by clicking on them.

Step 0: what tools you need – a long Philips (“cross”) screwdirever, preferably one with magnetic tip (but you can manage without it).

Step 1: disconnect the antenna from the wireless card. This is important, since it is connected to the LCD panel, which we need to remove. Do this by pulling carefully upwards on the connectors (not the wire). Don’t worry about knowing which wire goes where when reassembling, since it is clearly marked (with small white / black arrows).

Step 2: tilt the screen all the way backwards (so that it is parallel with the bottom part) and remove the upper part of the cover. There is a small opening where the marking is on the image, you can start there. Carefully remove the whole cover. It has a couple of plastic “ears” which you have to be careful not to break.

Step 3: remove the battery, hard drive, optical drive and bluetooth adapter. You eject the battery by sliding the middle lever. Remove the hard-disk by removing the two screws marked at the right. You can also remove the bluetooth adapter, which is near the harddisk. Sidenote: except the screws from the harddrive, you can distinguish the screws from the lower part and the upper part by their length. The rule is: lower part – long screws, upper part – short screws. To remove the optical drive, first remove the screw marked by a lock, and then push on he metal part with the screwdriver. This should pop it out just enough that you can pull on it.

Step 4: remove the screws holding the screen and the two screws holding the keyboard.

Step 5: disconnect the CMOS battery (this will result in you loosing your BIOS settings, which you will have to reset at the first boot after assembly). Also, disconnect the keyboard. This is a tricky connector: you have to flip the upper part open to remove the cable. Also, when putting it back, you first have to make sure that you’ve properly aligned the cable with the connector, and then push down on it. If it doesn’t go easy, don’t force it, rather take it out and try again, making sure that the alignment is correct (straight).

Step 6: disconnect the LCD panel and remove it. Unscrew the upper part, in the locations marked with “P”. Disconnect the two cables linking it to the mainboard (the ones towards the middle). Flip the base over and remove the bottom screws also. At this point you can separate the upper and lower part of the base.

Step 7: You can remove the PCMCIA adapter.

Step 8: The laptop is almost completely unassembled at this point. You can continue removing parts if you need to, however take care when working around the coolers: tightening them too much can result in the CPU/GPU cracking. Make them too loose however, and your cooling will suffer.

Happy hacking!

Watch out for those reviews…

Attila-Mihaly Balazs — Fri, 23 Oct 2009 17:46:00 +0300

Recently I was buying a notebook HDD, and after considering a Samsung SpinPoint model, I’ve looked around the net to see if there were any known issues with the model. So I stumbled upon this page and my blood ran cold. Quote:

One of the most common problems Samsung SpinPoint hard drives experience is burnt cuircuit board(PCB).

…

Samsung hard drives could also suffer from firmware problems.

…

Another quite common symptom Samsung drives experience is clicking/knocking sound.

…

There is one more problem that is typical for all hard drives and Samsung drives particularly: bad sectors.

Is this drive really of such poor quality? Does it really have all these problems? But then I started looking around on their site at they seem to have the same or very similar text for every type of HDD out there. The conclusion: they (Data Cent) are just trying to spam Google and I’m inclined to believe that most of their advice isn’t founded on facts, but rather on a randomized text generator. I for one encourage people not to take their business to such a company.

PS. All the links to them are nofollow, so I’m not giving them any Google love.

Picture taken from barnoid’s photostream with permission.

And now for some upbeat news

Attila-Mihaly Balazs — Fri, 23 Oct 2009 10:55:00 +0300

While I certainly like to rant, one shouldn’t forget about the more sunny side of life (unless you want to go berserk). So here are some random positive things:

Some songs which I like:

http://www.youtube.com/p/D984AB44FF987DDA&hl=en&fs=1 A funny image from [a friend](http://vtopan.wordpress.com): [![1940](http://lh3.ggpht.com/_hrvCBhtWhJ4/SuFhdS8Uo7I/AAAAAAAAB-M/7C_-IVmU1Q8/1940_thumb%5B3%5D.jpg?imgmax=800 “1940”)](http://lh5.ggpht.com/_hrvCBhtWhJ4/SuFhcyGYsmI/AAAAAAAAB-I/mLj5AzpNFRg/s1600-h/1940%5B5%5D.jpg) A couple of great freeware programs for the Windows platform: - [CDBurnerXP](http://cdburnerxp.se/) – does everything Nero does, for free! - [DVD Flick](http://www.dvdflick.net/) – while from the technical standpoint it is “just a wrapper” over FFMpeg and similar tools, it does a great job – you can create your DVD in a couple of steps - [foobar2000](http://www.foobar2000.org/) – a great little MP3 player, especially for those of us who liked the old Winamp, before it tried to do everything. At it can also do batch transcoding! - [IrfanView](http://www.irfanview.com/) – *the* free image viewer / converter! - [7-zip](http://www.7-zip.org/) – open source WinRar. Supports a lot of formats - [Far Manager Open Source](http://www.farmanager.com/download.php?l=en) – a great native win32 file manager with a retro look - [BB FlashBack Express](http://www.bbsoftware.co.uk/bbflashbackexpress/home.aspx) – a free screen capture software which works great - [VideoLan](http://www.videolan.org/) (or [VLC](http://www.videolan.org/) as it is better know) – the *simple* solution to play all your media, without having to install tones of codecs. If it would have a little better playlist management, I would use it as my primary media-player A great quote: “the difference between communism and capitalism is that in first men exploit other men, and in the second it is the other way around”. Found it via [this New York Times blog](http://freakonomics.blogs.nytimes.com/), written by the authors of [Freakonomics](http://hype-free.blogspot.com/2009/08/freakonomics-review.html) (and now the sequel []()[“\>’\>Superfreakonomics](“http://www.amazon.com/gp/product/0060889578?ie=UTF8&tag=hypefree-20&linkCode=as2&camp=1789&creative=390957&creativeASIN=0060889578”)). It is a great blog, worth the read. Where elsewhere do you find [a rigorous analysis of the logic in newspaper comics](http://freakonomics.blogs.nytimes.com/2009/10/07/freakonomics-quiz-doonesbury-logic/)? So there you have it, have a great day! An maybe listen to some [french striptease songs](http://vtopan.wordpress.com/2009/04/06/top-ten-striptease-songs/) :-) (just a little SEO for a friend ;-))

The difference between additive and subtractive color schemes

Attila-Mihaly Balazs — Thu, 22 Oct 2009 18:14:00 +0300

I’ve known for the longest time that there are two ways of creating / describing colors: additive (RGB) and subtractive (CMYK). However I never really understood the equivalence between them, until recently, when I picked up a book which presented the general concepts of typography. This is very cool, so I’ll try to document it here (because I didn’t find it elsewhere on the ‘net – although it almost certainly exists).

Lets begin with the additive part:

This is done by focusing one to three of the base colors (Red, Green and Blue) in the same point to create a given color. (Technically it isn’t “the same point”, just three points in such close proximity that you can’t tell the difference). The base colors can be generated in different ways: an electron beam hitting differently colored phosphor particles (CRT), colored LEDs or filtering out colors from a white light with colored polarization filters (LCDs). Already we can see a similarity between the “additive” and “subtractive” models: even when using the additive model, we sometimes create the light using subtraction (in the case of LCDs).

Now for the subtractive part:

Here we have a light source emitting white (ie. containing all three of the RGB components) beams of light. These beams hit a surface and some of the components get absorbed (“subtracted” from the beam), while others get reflected. The ones reflected form the resulting color. From this point of view the subtractive model is analogous to the way LCDs create colors. The three colors used in the subtractive model (Cyan, Magenta and Yellow) are chosen because they absorb the different components (ie R/G/B) of the white light.

There are other technical details (for example: the presence/absence of each component is not a binary 0/1 number, rather a continuously varying one, resulting in an infinite number or colors. or an other: the absorption in the case of the CMYK model is not perfect nor uniform between the different components. also, absorbing all the light – ie. creating black – is theoretically possible by combining CMY, but it would result in a – relatively speaking - very thick layer of paint, which is why black paint is also used in the printing process), but the essence of it is: additive and subtractive models are very similar.

Hope this helps somebody :-)

If you use Perl…

Attila-Mihaly Balazs — Thu, 22 Oct 2009 16:31:00 +0300

Please take a minute an answer the poll “Which editor(s) or IDE(s) are you using for Perl development?“ (via perlbuzz and szabgab).

Picture taken from Knightrider’s photostream with permission.

A couple of new challenges

Attila-Mihaly Balazs — Thu, 22 Oct 2009 15:05:00 +0300

Here are a couple of challenges I found on the interwebs:

SSHliders – from ethicalhacker.net. This one is centered around *nix shell scripting and more advanced topics like pipes.
Hugi Size Coding Compo #29 (from Hugi) – not much time left there, the deadline is the 28th of October. No flashy prizes either, just the bragging rights that you’ve created something useful in less than 124 bytes (the current leader)

Also, the solution to the Prison Break EH challenge has been posted (on EH.net with additional videos on the radajo blog). There is a lot of cool networking info in there, worth the read!

Have fun!

Using TarInputStream from Java

Attila-Mihaly Balazs — Wed, 21 Oct 2009 19:38:00 +0300

Recently I had to parse trough a bunch of logs, scattered in subdirectories and different types of archives (tar, bz and gz). My first thought was of course Perl (since it is the language for parsing quasi-freeform text), however I didn’t have “streaming” implementation for the archive modules, which in my case was very important, since the archives were big and reading them completely into memory was not acceptable. So I fund the TarInputStream / CBZip2InputStream from Apache Ant and GZIPInputStream, which is readily available in the JRE. While the last two are quite straight-forward to use, I’ve had to beat my head against the wall for quite some time before I managed to use TarInputStream. To save other people the hassle, here is a short writeup on what I’ve learned:

After creating the TarInputStream, you start out by calling getNextEntry.
You do this until it returns null (similar to how you would read a textfile line-by-line with BufferedReader)
tar doesn’t actually compress anything, it just concatenates the data in a sequence of

series. After calling getNextEntry the TarInputStream is positioned right at the start of the data for the given entry (if it is a file, which you should also check)
To read the data associated with the TarEntry you just obtained, you have two possibilities:
- You can use the the copyEntryContents method on the stream to put the data in an other stream (in memory, in an other file, etc). Just make sure that you have enough memory / disk space to do so
- You can read the contents directly from the stream. For example you can layer a GZIPInputStream (or CBZip2InputStream) over the TarInputStream if you have a gz / bz2 in a tar (usually it’s the reverse, this was the case for my little parser for example)

One thing to watch out for if you choose the second method, is the fact that TarInputStream is very sensible to positioning. So if the stream you layer on top of it has a off-by-one error (ie. it reads a couple of bytes more than the actual size of the data), you can quickly get a mysterious IOException which says something along the lines of “reading from output buffer”.

My solution to the problem was to layer a custom FilterStream on top of TarInputStream before handing it over to an other stream which does two things:

it makes sure that the stream on top of it can read only N bytes, where N is the size of the entry and
when close is called on it, it doesn’t propagate it to the TarInputStream (so that it doesn’t get closed and further entries can be processed)

Below you can see this filter stream:

public class SizeLimiterInputStream extends FilterInputStream {
  final long maxSize;
  final InputStream base;
  long alreadyRead;

  public SizeLimitInputStream(InputStream in, long maxSize) {
    super(in);
    this.maxSize = maxSize;
    alreadyRead = 0;
    base = in;
  }

  @Override
  public synchronized int available() throws IOException {      
    long a = base.available();
    if (alreadyRead + a > maxSize)
      a = maxSize - alreadyRead;
    return (int)a;          
  }

  @Override
  public void close() {
    // do nothing
  }

  @Override
  public boolean markSupported() {
    return false;
  }

  @Override
  public void mark(int readlimit) {
    // do nothing
  }

  @Override
  public void reset() throws IOException {
    // do nothing 
  }

  @Override
  public synchronized int read() throws IOException {
    if (alreadyRead >= maxSize)
      throw new EOFException();
    int r = base.read();
    alreadyRead += 1;
    return r;
  }

  @Override
  public synchronized int read(byte[] b) throws IOException {
    return read(b, 0, b.length);
  }

  @Override
  public synchronized int read(byte[] b, int off, int len) throws IOException {
    if (alreadyRead >= maxSize)
      return -1;
    if (alreadyRead + len > maxSize)
      len = (int)(maxSize - alreadyRead);
    int r = base.read(b, off, len);
    alreadyRead += r;
    return r;
  }

  @Override
  public synchronized long skip(long n) throws IOException {
    if (n < 0)
      return 0;
    if (alreadyRead >= maxSize)
      return 0;
    if (alreadyRead + n > maxSize)
      n = maxSize - alreadyRead;
    long r = base.skip(n);
    alreadyRead += r;
    return r;
  }

}

So your code would like something along the lines of:

TarInputStream tis = new TarInputStream(fileInputStream);
TarEntry tarEntry;
while ((tarEntry = tis.getNextEntry()) != null) {
  if (tarEntry.isDirectory()) continue;

  InputStream tmpIn = new SizeLimitInputStream(tis, tarEntry.getSize());                
  // process tmpIn - create other streams on top of it for example ...
}
tis.close();

Hope this helps.

Picture taken from quapan’s photostream with permission.

Avalonix Wireless Camera review

Attila-Mihaly Balazs — Tue, 13 Oct 2009 18:07:00 +0300

A wireless security camera is quite an interesting piece of technical equipment, which – the conventional wisdom holds – can deter people from breaking the rules (whatever those might be) or help after the fact demonstrate “who done it”. Their real value is however questionable. For one, they can become a prime target for attackers (“the first to fall”). Second of all, regardless of what is shown in TV shows, the resolution is quite poor, so it is improbable that you would get something out of it which could be used to identify a person with any certainty. Finally, there is a big problem with unauthorized people getting access to the pictures (voyeurism), especially with wireless security cameras, and especially with Internet connected wireless security cameras.

If you have carefully considered all the possible drawbacks and still want to install a wireless security camera (or more), you might want to take a look at the Avalonix 5.8GHz Wireless cameras. They advertise a reception distance between 3000ft (around one Km) and 7 miles (around 11 Km) with a proper antenna. They have a separate night vision mode and 420 lines of resolution. Again, consider if this is enough for your purpose. 420 lines is lower than the resolution of a standard-definition TV (which is 480). Two other concerns are the fact that the only provider for Avalonix cameras (security-cameras-cctv.com) seems to be offline currently (even though the server responds to pings, the webpage doesn’t load – tried from two geographically distant networks - and the Google cache shows a lot of MySQL errors) and the only mention I could find contains a complaint. The technical specification also fails to clarify if this is an outdoor or indoor camera.

On the bright side of this wireless security camera, the domain seems to be rather old (stable) and its owner doesn’t hide behind a proxy registration service (off topic: I too agree with the opinion that only private individuals should by allowed to use the domain-by-proxy type services, and that businesses should be needed to provide real verifiable contact information).

Currently my advice would be to go for a different type of camera, one which is more widely available and for which more reviews exists. Again, consider that such equipment might be a prime target for vandals, so you might be better off installing it in a covert location, and eventually providing some cheap fake camera lookalikes. Also, consider using a high resolution camera. One good test is to take a digital camera and shoot a couple of photos from the angle where you are considering to mount the camera (include some people in the photos near the perimeter of the view field). Then take the photo (which is probably of a high resolution 5+ megapixel) and shrink it down to whatever the camera specification says. See if you can still see enough details of the people in the picture (and consider that scaling down preserves more information that just plain-out taking the picture at the smaller resolution – so if you are unable to distinguish faces, you won’t be able to with the camera).

Full disclosure: this is a paid review from ReviewMe. Under the terms of the understanding I was not obligated to skew my viewpoint in any way (ie. only post positive facts).

Grooveshark

Attila-Mihaly Balazs — Fri, 09 Oct 2009 17:55:00 +0300

Disclaimer: I received no compensation for this review. All the opinions are my own.

There are a couple of actions you can do with music you have on your computer:

You can listen to an arbitrary song from your library
You can jump in the song you are listening to
You can share it with a friend
You can convert it and listen using an alternative format (ie. on your MP3 player, on your phone, in your car, etc)
You can edit it using a sound editor

Thus, any service which aims to replace your music library should provide as many of these options as possible. Recently I found out about Grooveshark which provides all but the last two of these actions for free! Additionally they have a very slick interface and you don’t even have to sign up to use the service!

Additionally they provide an “auto-play” feature which adds “similar” songs to your playlist and seems to work acceptably (is is no Pandora though). An other thing I appreciate is the clear revenue model: it is ad supported or you can become a premium member for as little as 3\$ / month and see no ads. I don’t know if the numbers are right, but they seem much more plausible than the one of Jango. I don’t want to dish them, but their lack of business model (no premium accounts!?) always concerned me and recently they seem to experiment with a lot of things (like taking away the possibility of directly playing a song and rather linking to Rhapsody) which is both annoying and concerning. My only concern is the fact that I found this site via the now defunct Seeqpod, which used (intentionally or unintentionally) publicly available MP3 files to power their service. While Grooveshark seems to use their own servers, rather than arbitrary hosts on the Internet, the file name quality suggests that these files were arbitrarily downloaded from different sources (public folders? P2P? – who knows), a practice which might get them in hot water with the RIAA, especially given that they seem to be based in the USA.

In conclusion: Grooveshark is great for now. Hopefully it will last in its current (or better) form for a long time. Below you can see an example of the embedded player:

http://listen.grooveshark.com/songWidget.swf

Update: check out the new Grooveshark interface! It is very nice. Also, I’m a VIP subscriber to Grooveshark now!

Picture taken from gonzalovalenzuela’s photostream with permission.

Sony Ericsson Satio

Attila-Mihaly Balazs — Fri, 09 Oct 2009 17:07:00 +0300

I admit: I didn’t jump on the “hip new phone with all kinds of bells and whistles” bandwagon yet. I’m very content with my small, sturdy Nokia. However one has to admit that there is a certain kind of wow effect when one looks at phones like the Sony Ericsson Satio. Even though I had some bad experience with Sony Ericsson phones in the past (in my opinion they are the “looks shiny but don’t expect it to last more than a year” category), this phone looks nice.

It has all the features one would expect from such a phone (quad-band, Wi-Fi, Bluetooth, GPS, USB, etc). Of course it has a full QWERTY keyboard with touch-screen and uses a standard stereo jack (this is big advantage and seems to be a standard feature for Sony Ericsson products). However the price seems to be a little too steep. I still can’t see real return-on-investment in buying a phone from this category. You could argue that it brings the functionality of a netbook, a digital camera, a media player and phone in one, but I still feel that it doesn’t do at least an average job of all of them (I imagine that it is a little too small for a netbook replacement and it doesn’t have optical zoom on the photography side – of course no consumer phone does – which makes the quality of the photos too low, even for an amateur like myself).

Full disclosure: this is a paid review from ReviewMe. Under the terms of the understanding I was not obligated to skew my viewpoint in any way (ie. only post positive facts).

LandAirSea Systems - GPS Tracking

Attila-Mihaly Balazs — Fri, 09 Oct 2009 15:44:00 +0300

GPS Tracking is a contentious issue, even more when it is forced on the subject. There are however situations where we can be legally obligated to subject ourselves to such tracking (for example when we are using the car of the employer or if our legal guardian wishes so). The LandAirSea company specializes in such tracking systems. They provide a variety of models, some with real-time tracking capabilities, others requiring the physical re-acquisition of the equipment for later analysis. For example we have the tracking key (available also on Amazon – disclosure: the link contains my affiliate ID), a very small form-factor solution with mostly positive reviews.

The company has been around since 1994, so, as the old cliché says, they must be doing something right. One positive aspect I observed is that tracking seems to be free for many models, there are no additional recurring costs besides the purchasing of the GPS. On the downside (if this is a concern for you), the real-time tracking seems to be limited to the mainland of USA. In conclusion: good company, nice offer and it is worth considering if you are shopping around for such equipments. One thing to keep in mind is that GPS units can be quite slow to acquire the signal in my experience (up to a couple of minutes if it isn’t moving, and even more if the unit is in motion).

Full disclosure: this is a paid review from ReviewMe. Under the terms of the understanding I was not obligated to skew my viewpoint in any way (ie. only post positive facts).

Fixing CVS annotate

Attila-Mihaly Balazs — Wed, 07 Oct 2009 17:19:00 +0300

Yes, some of us work on projects started almost a decade ago and as such we use CVS (yes, CVS has many limitations and yes, git is better – for a nice introduction see Randal Schwarz’s video about git), but migrating is not directly justifiable (it would involve: training IT staff to be able to maintain the repo, rewriting automation code which relies on CVS and training programmers – even though some of these could be postponed given that git contains a CVS bridge). Anyway, the problem which I faced was the following: cvs annotate only displays the first 8 characters of the username, which can be ambiguous if multiple people have similar usernames (which can easily happen if there is a convention like name.surname). Here is my solution to the problem: fetch the log for the file get the user associated whit each version (in the log CVS includes the full usernames). Then fetch the annotated version of the file and use the version to disambiguate the user. Here is some Perl code:

sub processAnnotations {
  my $fileName = shift;
  my ($cmdLine, $pid, %revisions);

  $cmdLine = "cvs -z9 log -N '$fileName'";
  $pid = open F, "$cmdLine |";
  my $rev;
  while () {
    $rev = $1 if (/^revision ([0-9\.]+)$/);
    $revisions{$rev} = $1 if (/^date:.*?author: (.*?);/);
  }
  close F;
  waitpid($pid, 0);

  $cmdLine = "cvs -z9 annotate '$fileName'";
  $pid = open F, "$cmdLine |";
  my @annFileLines;
  while () {
    if (/^(\d[0-9\.]+)(\s+)\(\S+ (.*)/s && exists $revisions{$1}) {
      $_ = "$1$2(" . $revisions{$1} . " $3";
    }
    push @annFileLines, $_;
  }  
  close F; 
  waitpid($pid, 0);

  return join('', @annFileLines);
}

PS. I verified in the CVS source that the output width for the author field is hardcoded:

           sprintf (buf, "%-12s (%-8.8s ",
                 prvers->version,
                 prvers->author);

Picture taken from Valeriana Solaris’ photostream with permission.

CBT Planet - PR Newswire

Attila-Mihaly Balazs — Wed, 07 Oct 2009 16:08:00 +0300

cbt planet recently announced their MCITP bootcamps for 2010. MCITP stands for Microsoft Certified IT Professional, and in good Microsoft tradition, it doesn’t stand for one certification, but rather a group of certifications. So we have (list taken from the Microsoft site):

MCITP: Enterprise Desktop Administrator 7
MCITP: Consumer Support Technician
MCITP: Enterprise Support Technician
MCITP: Enterprise Administrator
MCITP: Server Administrator
MCITP: Database Administrator 2008
MCITP: Database Developer 2008
MCITP: Business Intelligence Developer 2008
MCITP: Database Administrator
MCITP: Database Developer
MCITP: Business Intelligence Developer
MCITP: Enterprise Project Management with Microsoft Office Project Server 2007
MCITP: Enterprise Messaging Administrator

Again, as with other Microsoft exams, each certification is composed out of two to three exams. There is absolutely no overlap between the exams necessary for the different types of certifications, which is somewhat surprising to me (I’ve briefly looked at the MS certs for developers and I seem to remember that there is more overlap there – but it is possible that this is representative for certification structure for the IT part of MS).

From what I’ve seen on their site, CBT Planet focuses mainly on providing online, self-paced study possibilities, but it seems that they are trying to enter into the bootcamp market. As a general rule, I don’t see the value in bootcamps. The two exceptions would be: if you know that it is absolutely necessary for the job you are applying for or if your employer is paying for it (in which case it can be considered an almost – but – not – quite – vacation). From my experience (based on participations in a couple of certification programs – no bootcamps though), anyone somewhat familiar with the subject can pass the final exams. If you go to the bootcamp: good luck with the exam!

Full disclosure: this is a paid review from ReviewMe. Under the terms of the understanding I was not obligated to skew my viewpoint in any way (ie. only post positive facts).

cbt planet

Attila-Mihaly Balazs — Wed, 07 Oct 2009 15:04:00 +0300

The issue of certifications is a contentious one in the field of IT. On one hand there are the people who try to hire without having necessarily the expertise to judge the candidate (ie. HR). On the other hand there are the people who complain about the low quality of “certified” people.

The best attitude I know about towards certification is the following (unfortunately I can’t recall where I originally read it): recognize that there are multiple type of job roles. If you are looking for someone axed exactly on one technology (ie. managing Solaris servers), a certification can be a useful indicator. However if you are looking for someone whose responsibilities are not so clear-cut (ie. Java developer, but also should know some Linux, Bash scripting and Perl if possible), you are much better of searching for people with certain personality traits than with these exact certifications. Again, something I’ve read and I agree wholeheartedly with: “recruiters don’t invent candidates – they find existing ones, if the exists”.

An other aspect of certificates is cost – both monetary and time. Because of this, it is usually a nice reward companies can give their developers.

Having said that, let me write a little about the MCITP (Microsoft Certified IT Professional) bootcamps announced by cbt planet. From what I’ve seen, CBT Planet focuses on providing on-line (ie. video) training for certifications. This is an easy and time-effective way to gather the information you need to pass the exam. The bootcamp is simply a condensed version of that.

The structure of the Microsoft certifications is as confusing as ever: as with many others, there is no “one” MCITP, rather there are many flavors. So, should you get it? It is a matter of personal preference. My personal opinion would be: if it is offered by the company, definitely. An other definitive yes is the case you know for sure that it is a requirement for a future job. Otherwise – not so much.

Full disclosure: this is a paid review from ReviewMe. Under the terms of the understanding I was not obligated to skew my viewpoint in any way (ie. only post positive facts).

One more thing…

Attila-Mihaly Balazs — Wed, 07 Oct 2009 13:36:00 +0300

So, if I started ranting on Microsoft, here is one more thing: you should never ever use Microsoft servers if you want to scale. The reasons is simple: currently the best scaling method is horizontal (ie. buy loads of cheap hardware and distribute the load between them). Using Microsoft server software would mean that for each server you would need to buy at least one license (or more, like in the case of MS Windows + MS SQL Server). This can easily cost you almost as much (if not more) than the hardware itself.

As far as I’m aware, only Microsoft and hosting providers (which recoup the cost from their clients) are running MS servers in large numbers. This is sustained by a quick check of the top \~5000 sites (as defined by Alexa) – less than 16% of them run IIS. To paraphrase Eminem:

Be smart, don’t be a retard! You take advice from someone who run their front-end webserver tier on VM’s, even though it makes no sense from a technology standpoint to do so?

PS. Yeah, there is the BizPark program from MS for startups – but we yet have to see the first large-scale success emerge from it (BTW, one of my favorite site – StackOverflow – runs Windows Server trough the BizPark program – and even they now use a Linux VM!).

Picture taken from mark sebastian’s photostream with permission.

My opinion about Microsoft, software piracy and everything

Attila-Mihaly Balazs — Mon, 05 Oct 2009 16:08:00 +0300

This post is a response to a blogpost on tudor g’s blog about software piracy issues in Romania, and as such it might not be of interest to you, dear international reader. If this is the case, feel free to skip this post.

Disclaimer: arguments are very emotional things. As much as we would like to think that they consists of logical statements and counter statements with the “best” arguments winning, in real life the acceptance or rejection of a given argument very much depends on the frame of reference of individual persons. With this in mind, I believe that Tudor and me have very different frames of reference (him being a Microsoft employee and me being an open-source enthusiast) and as such I’m quite sure that nothing written by me here will change his mind (and conversely, nothing written by him will change my mind). Still, I think that this is a useful exercise to get things off my chest and to document arguments for more open-minded people :-)

His first argument is that installing and using pirated software is harder that legitimate software because mechanisms like WGA - I don’t think that this argument holds water, since most people (the “average user”) can’t install an OS, regardless if it is pirated or legitimate. Just ask yourself the following question: how many of the non-technical people you know installed themselves the OS on their computer? I would bet you that the number is very, very close to 0.

If the OS is installed by the “neighbor kid from the 2nd floor”, then this argument doesn’t hold. Even more, many geeks pride themselves with being able to perform complicated tasks, like disabling the WGA, and as such, for them the existence of protection element is a positive thing (a challenge to be solved).

Finally, the inverse of the argument (that legitimate MS software is easier to use than pirated one) isn’t true either in my experience. I had numerous occasions where (completely legitimate, bought with the computer OEM) Windows failed to validate, a (again, completely legit, boxed version) Win 2k3 SBS suddenly refused to work because it needed to be a DC (and it told me after 3 months!), the Windows 7 beta deactivated itself periodically, VM’s deactivated themselves after moving from one machine to the other for purely technical reasons (even though the one-machine / one-owner / one-copy rule was always observed), etc.
The second argument is that there is no peer-pressure to pirate in Romania (that not “everybody is doing it”) – I would suggest him to visit any campus in Romania and check out the (pirated) software which can be found on the network. And not only that, but music, movies, books, etc. Or to go to repair shops and ask for MS Windows / MS Office being preinstalled on the computer – the answer will almost always be positive. Even more, the next generation feels entitled to these freebies (and it isn’t something specific to Romania either, thanks to the abundance of the freemium business model on the Internet).

In the long term (IMHO) less and less people will be willing to pay for things which they perceive as basic needs. The only options for old-style software companies (like Microsoft) are to include more and more technical measures to try to prevent this (even though the current measures already make MS Windows annoying to use) or to raise the level of punishment associated with piracy (which shouldn’t be possible in democratic countries because of public backlash)
Piracy doesn’t help the software companies by making their product more well known – if this would to be true, why do you think that there are associations in people’s mind like Microsoft – Windows, Office – Microsoft, image editing – Photoshop, CD burning – Nero and so on? Most people use whatever is already installed on the computer to accomplish their jobs. This is why OEMs get big bucks from software companies to preinstall their product.

I don’t buy the “starving programmer” argument either. The cost of copying software is minuscule. Which means that over 80% (this is of course a number pulled out of my rear, but I’m quite sure that the real number is somewhere in the ballpark) of each sale is pure profit. Which means that (a) a freemium type model can easily be sustained and (b) that even a few sales mean that the company makes a nice profit, and excessive focus on this part (ie. “we are loosing X billion of USD to piracy” – which BTW is not true for at least two reasons – first because the method to determine the number of pirated copies is questionable and second, because it assumes that every person who “pirates” would buy the product if s/he didn’t have access to the pirated version) is pure greed – which, let us remember, is one of the seven deadly sins.

Also, as a programmer, you don’t have to write commodity software. Let me tell you, there is very good money to be made from writing custom software for a small number of clients.

A final point I would like to make with regards the relation of piracy to innovation: remember that all three “big” powers (USA, Russia and China) started out (and some are still) by rejecting patents to bootstrap their industry. Something worth thinking about…
That people buy because somehow they are convinced that it is “the right thing”, not because of fear – I’m not seeing it. At least at the individual level I don’t know anybody who bought a single Microsoft product (including myself, I’m living off my MSDN AA licenses). At a company level the motivation (arguably) is mostly fear. They buy licenses for the same reason they pay taxes. Also (as I’ve already said at point 2), the willingness of people will only go down, not up.
Software is not overpriced, especially when considering the income level of Romanians – this is IMHO the best example for the “pink sunglasses” Tudor is wearing and how his frame of reference distorts his perception. The average (net) income per month in Romania for 2008 was somewhere around 400 USD (this would mean 4800 USD per year for those of you who use this frame of reference). Given this figure, is it reasonable to expect that people give more than half of their monthly income (or even all it, if we consider that a computer would need MS Windows + MS Office + AV) for software? May I venture a guess that (a) Tudor has at least five time the average net income and (b) he has free access to all of Microsoft’s software, and as such, he might not see the real situation? My challenge to Tudor would be: how much of the software he has right now on his personal machine did he pay for?

My conclusion is that software piracy is here to stay. Especially in more poor countries like ours. To give Microsoft what is due: they do really excellent software (not that they don’t do mistakes, like Vista – which is abysmal – I’m speaking from a first hand experience, having played with it on two “Vista certified” laptops and in VMs). Even so, their expectation is unrealistic at a minimum and even unethical. Also, as a developer, if you develop using a technology (OS, libraries, components, etc) for which you don’t have the source code, you will hit “undebuggable” issues sooner or later.

PS. Vista is the new ME – just worse:

Update: fixed some typos and errors in expression – thank you to my dear readers.

Space Invaders Rulz!

Attila-Mihaly Balazs — Mon, 05 Oct 2009 14:04:00 +0300

Found this in the mall. While the quality is somewhat poor and I’m not quite sure that the proper license fees were paid for the images (“raiders invasion” ???), it is still fun.

Two new challenges

Attila-Mihaly Balazs — Fri, 02 Oct 2009 16:47:00 +0300

Well, new for me at least…

The first one is 0x41414141.com. Just go to the site and you can start directly. As far as I know, this is not time-bound.

The second one is spargecoduasta.com (“break this code”). It is put up by BitDefender and I don’t know if it has a time limit. The levels I’ve seen seem to focus on C/C++. It is available in both Romanian and English.

Finally, a little off-topic, but still a challenge: The Science Knowledge Quiz – with the tagline “Are you more science-savvy than the average American?”. Via Pat’s Daily Grind (I’ve got 11 out of the 12).

Have fun!

VoIPstreet Affiliate Program review

Attila-Mihaly Balazs — Fri, 02 Oct 2009 16:35:00 +0300

Branded affiliate programs are a great way to offer services to your clients which are not your core competency (for example you are web-design company but you would also like to offer hosting to your clients so that they have a one-stop-solution experience). This is what VoIPstreet Affiliate Program does for VoIP: you can resell a service to your customers which is fully branded with your logo. The service provides unlimited calls in the USA and all the features you would expect from a PBX (like caller-id, transfers, on-hold music, etc). Other information you might be interested in are the prices and the available phone models. One thing which I couldn’t find are the financial details of the affiliate relation.

Are there any good? From searching around the interwebs I found some old (2007) complaints, but it seems that it was a problem with their data-center and I couldn’t find newer complaints (which is a good sign). They also take part in industry events like IT Expo, whish means that they are definitely not a fly-by-night operation (one has to be careful, since the VoIP space has a lot of fraudsters who use cracked PBXs to offer cheap services).

Full disclosure: this is a paid review from ReviewMe. Under the terms of the understanding I was not obligated to skew my viewpoint in any way (ie. only post positive facts).

SMOG button removed!

Attila-Mihaly Balazs — Thu, 01 Oct 2009 17:28:00 +0300

Almost a year ago I added a SMOG button to each blogpost, which (in a more or less serious manner) evaluated the “reading level” needed to understand the blogpost. However, today the site used for this service came up with a warning from Google saying that it might be malicious. I’ve looked into it, and indeed, it contains an IFRAME pointing towards a malicious site.

So I’ve taken down the script until this issue is resolved to protect people. Hopefully this issue will quickly be resolved.

Picture taken from riNux’s photostream with permission.

ERR – Exponentially Expanding Rants

Attila-Mihaly Balazs — Tue, 29 Sep 2009 17:51:00 +0300

I’m expanding! Not only am I ranting on my own soap box (blog), now you can read my rants on other blogs as well: fudsec just published my Knowing Walls from Speed Bumps rant. Thank you! If you are fed up with FUD in the security industry, check out the site and went away! Who knows, maybe we can even change a thing or two (yeah, righ!).

Thanks again to the fudsec guys for publishing my “article”.

Picture taken from Serendigity’s photostream with permission.

The Myths of Innovation

Attila-Mihaly Balazs — Tue, 29 Sep 2009 17:36:00 +0300

Some time ago I ranted about cognitive quantum leaps. Below is a presentation given at Google by Scott Berkun about the same topic. While the talk itself is two years old, the examples are still very relevant, and he presents it very eloquently. Enjoy!

http://www.youtube-nocookie.com/v/m6gaj6huCp0&hl=en&fs=1&color1=0x234900&color2=0x4e9e00

Carcassonne – a great boardgame

Attila-Mihaly Balazs — Tue, 29 Sep 2009 16:19:00 +0300

My personal favorites are (board)games which create complexity from simple rules, for example chess, go or FPSs in the virtual world. A boardgame which comes close to this ideal is Carcassonne. The rules can be explained in less than two minutes and it is great fun! See below the recording of a nine hour game:

http://www.youtube-nocookie.com/v/UsvNgtBWZeI&hl=en&fs=1&color1=0x234900&color2=0x4e9e00 Don’t be afraid though, that one was played using tiles from 3 sets. A normal game lasts around 30 minutes. You can [buy the game on Amazon](http://www.amazon.com/gp/product/B00005UNAX?ie=UTF8&tag=hypefree-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=B00005UNAX) (disclosure: the link contains my affiliate ID) or, if you are in Romania, you should check out [cutia.ro](http://cutia.ro/Hans-im-Gluck/Regulamente/Carcassonne.html). Have fun! PS. There seemed to have been [a site where you could play it online](http://games.asobrain.com/), but it seems to be offline. There is [some speculation](http://answers.yahoo.com/question/index?qid=20090920121644AAQYJmM) that this is due to copyright violation on the part of the site and/or due to the site being hacked.

Network Forensics Contest submission

Attila-Mihaly Balazs — Tue, 29 Sep 2009 13:26:00 +0300

Some time ago I mentioned the Network Forensics Puzzle. The contest is now over and since I didn’t win, I’ll publish my submission below – it was after all correct, but not quite what the judges were looking for (congratulation to the winner).

After validating that the MD5 sum for the downloaded file matches the one specified on the website, I first opened it up in NetworkMiner (http://networkminer.sourceforge.net/). I find the overview it gives much easier to understand than the statistics provided by Wireshark. Using it I identified the data stream between Ann’s computer and the unidentified laptop.

1. What is the name of Ann’s IM buddy?
Sec558user1 - this is tricky because the IM (which seems to be AOL - but many other IM’s behave in a similar fashion) routes chat traffic trough central servers (64.12.24.50 in this case - which belongs to AOL, making it even more probable that AIM was used) to make NAT traversal a non-issue, while file transfers are done trough direct connection to conserve bandwidth.

2. What was the first comment in the captured IM conversation?
Here’s the secret recipe… I just downloaded it from the file server. Just copy to a thumb drive and you’re good to go >:-)
(actually, > is escaped as HTML - ie >)

3. What is the name of the file Ann transferred?
recipe.docx

4. What is the magic number of the file you want to extract (first four bytes)?
50 4B 03 04 - Which corresponds to PK…, signaling that we are potentially dealing with a ZIP archive here. This is further reinforced by the filename (.docx, which is the new “open” document format from Microsoft - basically, it consists out of a zipped XML - similarly to the OpenOffice.org format)

5. What was the MD5sum of the file?
8350582774e1d4dbe1d61d64c89e0ea1

This is again tricky, because ZIP (like many other formats) admit arbitrary data after the logical end of the file. So, using a hex editor, we first carve the the part starting at PK in the 192.168.1.158 -> 192.168.1.159 (be careful not to include the traffic in the reverse direction). Then we need to convince ourselves that the end of the file has been correctly identified at the byte level. To do this we could study the ZIP specification (http://www.pkware.com/index.php?option=com_content&task;=view&id;=64&Itemid;=107) or use a more empirical level: using a hex editor (HxD for example - http://mh-nexus.de/en/hxd/) eliminate the last byte of the file and “test” the integrity of the file (using the Test option from 7-zip for example - http://www.7-zip.org/ - but one could use almost any de-archiving program, since almost all of them offer a “Test” option). The test will fail. Now add back the last byte (which is 0x00) and perform the test again. It will succeeded. This means with a big probability that we correctly identified the actual (logical) end of the file.

6. What is the secret recipe?
The most recent version of OpenOffice.org (3.1.x) can open the docx format, so the following can be retrieved on any platform, regardless of whether MS Office 2007 is installed (an alternative solution would be to use the free MS Word 2007 viewer or the import filters available for older versions of MS Office).

The contents (sans the formatting):
Recipe for Disaster:
1 serving
Ingredients:
4 cups sugar
2 cups water
In a medium saucepan, bring the water to a boil. Add sugar. Stir gently over low heat until sugar is fully dissolved. Remove the saucepan from heat. Allow to cool completely. Pour into gas tank. Repeat as necessary.

Back with a vengeance

Attila-Mihaly Balazs — Tue, 29 Sep 2009 12:22:00 +0300

I’ve had limited connectivity / time in the last couple of weeks, but I’m back baby!

The lesson of the day: you shouldn’t let your AV vendor provide general security. The lesson comes to us via Graham Cluley’s Sophos blog, which informs us that Sophos released a free encryption tool. So, what’s wrong with it?

It uses symmetric crypto! Let me repeat that for you: it uses symmetric cryptography and recommends that you communicate the encryption password to the counterparty via an other communication channel. So much for ease of use and the asynchronous nature of email!
The feature sheet starts out by declaring that PKI has failed us, and then it touts a proprietary centralized solution for managing passwords.

In conclusion: this is a software which shouldn’t have been made. It is simplistic, cumbersome to use and provides no added benefit over using something like 7-zip with encryption (which also uses AES, also does compression – most probably better compression – and it to can create SFX password protected archives).

Go and install Thunderbird with Enigmail.

For added fun: the brochure says “Protects against “brute force” attacks by increasing response times each time the user enters the wrong password”, and they then proceed to give you a CLI and COM interface to the program which can be used directly to bruteforce the archives and without these timeouts. Nice!

Ethical Hacker challenge “Prison Break” solution

Attila-Mihaly Balazs — Wed, 02 Sep 2009 15:44:00 +0300

As I usually do, I’ll publish my entry for the Ethical Hacker challenge after the deadline passed:

Challenge Question 1: What is the most probable reason Michael could not get network connectivity from the desk Ethernet jack? What actions should the team take to determine exactly what is going on, collect full traffic captures, and gain full access to the network?

Most probably the switch to which the given port is connected has MAC address filtering turned on. To circumvent this, they must clone the MAC address of the VOIP phone.

The easiest way to do this is to start capturing the traffic on the network interface of the laptop and then plug the VOIP phone into it. The initial packets (most probably DHCP requests) will reveal the phone MAC address. Sidenote: most ethernet ports these days are auto-sensing (ie. no crossover cable is required). But just to make sure, one should use a crossover cable or an intermediate switch (not hub!) is one is available. After the MAC address has been determined, the host OS should be instructed to use the given MAC address for the laptop network card. You can find instructions for Linux here: http://linuxhelp.blogspot.com/2005/09/how-to-change-mac-address-of-your.html and for Windows here: http://www.irongeek.com/i.php?page=security/changemac

Sidenote: given that the packet captures show two distinct networks (192.168.1.0/24 and 172.29.0.0/16), it is clear that the administrators have tried to separate the computer networks from the VOIP one. However, relying only on different (sub-)nets is extremely weak and at least VLAN level separation should have been implemented (then again, maybe the available switches don’t have VLAN features). 172.29.0.0/16 most probably is the VOIP network, since we see SIP packets on it and 192.168.1.0/24 the computer network.

If only MAC filtering is implemented, after changing the MAC address, it is possible to join any of the two available networks, meaning that they can interact with the “computer” network, even if the given port was originally assigned to a VOIP phone.

Challenge Question 2: What tool should Lincoln download, if any, to be able to capture traffic on the desktop computer?

Sectools.org contains a nice list of available packet sniffers ( http://sectools.org/sniffers.html ). Given the constraints, my tool of choice would be WinDump, the Windows port of tcpdump ( http://www.winpcap.org/windump/install/ )

Challenge Question 3: Starting with the reverse connection from the desktop computer, describe a step-by-step approach that could be applied prior to 09:00 the next day in order to capture the network traffic on the remote network and get a capture file for further in-depth analysis. Make sure your approach follows Michael’s advice to avoid detection.

download WinDump ( http://www.winpcap.org/windump/install/bin/windump_3_9_5/WinDump.exe ) and WinPcap to the laptop
use the instructions provided at the following link to construct a portable version of WinPcap: http://paperlined.org/apps/wireshark/winpcap_silent_install.html
you can package up all the files (WinDump and the WinPcap DLLs + driver) into a single file using the SFX functionality from 7zip. To make sure that you don’t get under the 0.5 meg limit, use Zip with the Store algorithm
upload the resulting file to the general’s desktop (this part of the challenge is a little forced IMHO, since the IDS should have detected the reverse connection if it is sensible to long-lived, low traffic connections…)
launch the SFX, wait until all the files are extracted and copy npf.sys into c:\Windows\System32\drivers
before 9:00 AM (at 8:55 for example) launch WinDump (this will capture at most 5 MB of data):
```
WinDump -i 1 -w capture.pcap -C 5
```
after WinDump has stopped, retrieve the capture file and clean up (delete the driver, the SFX file, etc)

Challenge Question 4: Help the team complete this aspect of their mission by analyzing the packet capture file collected on the desktop computer and provide detailed information about the environment. Your response should at least include the type of network traffic collected, details about the General’s laptop computer, details about the Scylla Codes server plus any other server available, and provide the names and contents of the files stored on the server the input passphrase is based on.

The collected traffic consists of 6 requests made to the Scylla server (10.10.20.94) using HTTPS. To decode them, first convert the provided key file into PEM format with OpenSSL:

openssl rsa -in server.key -out server_key.pem

Then use the resulting PEM file as described here for example to let Wireshark decode the traffic: http://www.novell.com/coolsolutions/appnote/19321.html

Now you can use the “Follow SSL stream” functionality from Wireshark to analyze each request. From the headers it seems that the general’s laptop is running Windows Vista Media Center (tablet? edition), while the Scylla server is running Linux/Apache:

User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506) Server: Apache/2.2.8 (Unix) mod_ssl/2.2.8 OpenSSL/0.9.8g DAV/2 PHP/5.2.9

Challenge Question 5: What are the validation code and input passphrase used by the General to generate the Scylla validation code for this week?

The validation code is “6189db841f01413a05a53b7135137a17”

BONUS QUESTION: Briefly describe your recommendations about how The Company could have detected and defended against the tactics you described in your answer to Question 3.

The attack could have been prevented by using a whitelisting product which doesn’t let unknown executables be started. Other mitigating measures would be:

using Group Policy to disable autorun
using Group Policy to controls what kind of USB devices can be used: http://www.windowsecurity.com/articles/Control-USB-Devices-Group-Policy.html
using a switched network
using an outbound proxy and make sure that only valid HTTP/HTTPS traffic can leave the network

One could work around many of these restrictions (for example: finding a vulnerability in an installed software, running meterpreter in-process, killing the whitelisting software, masking the outbound connection as a HTTP one, using ARP spoofing to get around the switched network, etc), but it raises the bar considerably.

Geek pr0n – time lapse video of building racks

Attila-Mihaly Balazs — Mon, 31 Aug 2009 18:13:00 +0300

[youtube=http://www.youtube.com/watch?v=aOnNpBkRam0] Via [run-virtual.com](http://www.run-virtual.com/?p=346).

T2‘09 Challenge

Attila-Mihaly Balazs — Mon, 31 Aug 2009 18:00:00 +0300

Sorry for being a little late: the T2’09 challenge just started. Via the F-Secure weblog. Don’t be fooled by the fact that page already contains two entries (“Mr. Speed” and “Mr. Style”) in the top. From what I understand, these are to signal that two winners will be selected, one for speed and one for style.

The page also contains entries from past years for you to play with.

Careful with that axe^H^H^H static, Eugene!

Attila-Mihaly Balazs — Mon, 31 Aug 2009 17:36:00 +0300

An other instance from the “bugs which will bite you” series:

public class TestStatic {
    static class Foo {
        static Foo instance = new Foo();
        static String name = Foo.class.getName();

        public Foo() {
            System.err.println("Hello, my name is " + name);
        }
    }

    public static void main(String[] args) {
        System.err.println("Your name is what?\n"
            + "Your name is who?\n");

        new Foo();
    }

}

Can you spot the bug? Hint, here is the output:

Your name is what? Your name is who?

Hello, my name is null
Hello, my name is TestStatic$Foo

A final hint: here is what FindBugs reports on the third line: SI_INSTANCE_BEFORE_FINALS_ASSIGNED

Yet an other reason to reduce your FindBugs count to zero before releasing software!

Picture taken from helena.40proof’s photostream with permission.

Freakonomics review

Attila-Mihaly Balazs — Sat, 29 Aug 2009 18:02:00 +0300

[![](http://2.bp.blogspot.com/_hrvCBhtWhJ4/Spk7R7VvqjI/AAAAAAAABMk/A7kIaVald-w/s320/51YA2F7-dPL._SL160_.jpg)](http://www.amazon.com/gp/product/0060731338?ie=UTF8&tag=hypefree-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=0060731338)

I know, I know, I’m quite late to the game (Freakonomics was first published in 2005), but I still feel that this book deserves a review.

What I like about it the most, is the fact that it tries to teach critical thinking, a thing which is lacking these days. The book provides vivid and easy to read descriptions of investigative stories, similar to detective/murder-mystery books, about how seemingly distant events relate to one-another. Each chapter stands on its own and can be read-trough in a short amount of time. Where necessary, the book provides high-level description of the statistical terms (like regression-analysis), without giving the impression of “talking down”.

My favorite quote from the book is actually not from the book authors, but non-the-less I find it very fitting (and I’m sure that the authors agree with me, since they included it in the book :-)):

“We Associate truth with convenience,” he wrote, “with what closely accords with self-interest and personal well-being or promises best to avoid awkward effort or unwelcome dislocation of life. We also find highly acceptable what contributes the most to self-esteem.” Economic and social behaviors, Galbraith continued, “are complex, and to comprehend their character is mentally tiring. Therefore we adhere, as though to a raft, to those ideas which represent our understanding.”

I consider this book one worth reading. The only negative was that I got the “extended” version (which is not the one I link to) and I found that the “extension” was just some rehashing of the content from the book and from their blog. So get the real thing.

PS. Wikipedia says (so it must be true :-p) that a sequel is coming. I’m looking forward to it! Even if it will be half of the quality of the original (as sequels usually are), it should still be a great read!

Full disclosure: the links include my Amazon Affiliate ID.

Spot the error

Attila-Mihaly Balazs — Fri, 28 Aug 2009 16:08:00 +0300

C++ compilers are notorious for giving error messages which point you in the wrong direction. However even simpler languages can have issues. Can you spot the real problem with the java code below?

There is a comma missing between the parameters! Nice, ey? (To be fair, on the sidebar Eclipse shows two errors, one of which correctly identifies the problem, but I still scratched by head for a few moments).

Perl is everywhere!

Attila-Mihaly Balazs — Thu, 27 Aug 2009 18:04:00 +0300

[youtube=http://www.youtube.com/watch?v=04YipMChphk]

Something which is not appreciated enough IMHO is just how much of the interwebs runs on Perl: for example Frozen Bubble is written in Perl. Also, from some error messages I’ve got the impression Yahoo Pipes uses (is written in?) Perl.

And just before you accuse me of being a Perl fanboy (which I am BTW :-p), here is a fascinating document: Far More Than Everything You’ve Ever Wanted to Know about.

Youtube gadget generator

Attila-Mihaly Balazs — Thu, 27 Aug 2009 17:43:00 +0300

Some time ago I posted about how the Google Gadget code for Youtube seems to be borked up. Now it seems that they completely removed the option from the YouTube pages, for whatever reason, but the old code still seems functional. So below you can find a small Javascript which generates the equivalent code for a given Youtube user (if you are reading this in an RSS reader, you need to click trough to the post page, since chances are that the reader stripped out the JS for security reasons).

Of course this is an unofficial and unsupported solution, so it might break at any time and without warning!

Grey Panther's Place

An interesting proof for Pythagoras’s theorem

Finding the N-th word in a complete dictionary

Problem statement

Some initial observations

Finding the index of a string

Finding the N-th string

Can we do it simpler?

More speculation

Source code

The limits of science

The core principles of science

Science is not “truth”

Science is not math

Other limitations of science

Words about human-centric fields

A fresh start with Pelican

On benchmarks

Numbers every programmer should know and their impact on benchmarks

Proxying pypi / npm / etc for fun and profit!

Programming advent calendars for 2013

Cleaning up Google AppEngine Mapreduce Jobs

Capturing your screen on Ubuntu - with sound

Tips for running SonarQube on large / legacy codebases

Nested fluent builders

Passing UTF-8 trough HTTP

Connecting to the MtGox market data feed using Perl

Converting datetime to UTC in python

Recovering your RoTLD password for domains registered trough Gandi.NET

Free software for Windows

What every programmer should know about X

Ensuring the order of execution for tasks

Java Runtime options

Changes to String.substring in Java 7

(Re)Start me up!

Upgrading from MySQL to MariaDB on Ubuntu

Cluj-Napoca (Romania) wins the title of European Youth Capital 2015

Writing beautiful code - not just for the aesthetic value

Example 1: Double Trouble

Example 2: Where is my null at?

Example 3: We come up empty

Example 4: Careful with that static, Eugene!

Example 5: Remove old cruft

Conclusion

A (mostly) cross-platform clickable map of Romanian counties

Every end is a new beginning

Proxying URL fetch requests from the Google App Engine

Helper for testing multi-threaded programs in Java

GeekMeet talk about Google App Engine

Lightning talk at Cluj.PM

Running pep8 and pylint programatically

Clearing your Google App Engine datastore

Relaxed JSON parsing

Updating the root certificates for Java

Vagrant and VirtualBox on Windows

Another friend blogging

Using Jython from Maven

How to post high-quality videos to Google Video

Integrating Maven with Ivy

Upgrading the Options (GlobeTrotter) GI515m

Getting the most out of your audio recording with Audacity

More videos

Of maps and men

Informative videos about copyright and remixing issues

Recording test performance with Jenkins

Audio quality redux

Power Line Humm Removal With Audacity

Protein Shakes site review

Running JRuby on 64 bit Windows

Using less with syntax highlight

Link love

100 years of style

Quick’n’dirty Mediawiki file crawler

Creating a non-MAC bound CentOS 6 machine

Levant Digital Marketing review

Paving site review

IVA site review

Setting up git-daemon under Ubuntu

Adding tab completition to Maven3 under Ubuntu

adulttoys.com review