LiraNuna's Development Blog http://www.liranuna.com Just another coder Sun, 03 Mar 2013 21:01:52 +0000 en-US hourly 1 http://wordpress.org/?v=3.5.1 PHP Path Resolution – Now PHP 5.3 compatible! http://www.liranuna.com/php-path-resolution-now-php-5-3-compatible/ http://www.liranuna.com/php-path-resolution-now-php-5-3-compatible/#comments Sun, 29 Aug 2010 07:52:40 +0000 LiraNuna http://www.liranuna.com/?p=1125 Presenting the PHP 5.3 compatible version of my PHP path resolution class!

Why there should be a difference? Because my original version used create_function, which… created a whole new function object every time you used one of the class’s methods. This was very memory expensive and even incurred a slight performance hit.

The new version uses PHP 5.3′s anonymous functions to create more readable code in addition to reducing memory consumption and execution time.

As usual, the source code is under the WTFPL for you to enjoy without any restrictions.

<?php
 
/**
 * @class Path
 *
 * @brief Utility class that handles file and directory pathes
 *
 * This class handles basic important operations done to file system paths.
 * It safely renders relative pathes and removes all ambiguity from a relative path.
 *
 * @author Liran Nuna
 */
final class Path
{
	/**
	 * Returns the parent path of this path.
	 * "/path/to/directory" will return "/path/to"
	 *
	 * @arg $path	The path to retrieve the parent path from
	 */
	public static function dirname($path) {
		return dirname(self::normalize($path));
	}
 
	/**
	 * Returns the last item on the path.
	 * "/path/to/directory" will return "directory"
	 *
	 * @arg $path	The path to retrieve the base from
	 */
	public static function basename($path) {
		return basename(self::normalize($path));
	}
 
	/**
	 * Normalizes the path for safe usage
	 * This function does several operations to the given path:
	 *   * Removes unnecessary slashes (///path//to/////directory////)
	 *   * Removes current directory references (/path/././to/./directory/./././)
	 *   * Renders relative pathes (/path/from/../to/somewhere/in/../../directory)
	 *
	 * @arg $path	The path to normalize
	 */
	public static function normalize($path) {
		return array_reduce(explode('/', $path), function($a, $b) {
			if($a === 0)
				$a = "/";
 
			if($b === "" || $b === ".")
				return $a;
 
			if($b === "..")
				return dirname($a);
 
			return preg_replace("/\/+/", "/", "$a/$b");
		}, 0);
	}
 
	/**
	 * Combines a list of pathes to one safe path
	 *
	 * @arg $root	The path or array with values to combine into a single path
	 * @arg ...		Relative pathes to root or arrays
	 *
	 * @note		This function works with multi-dimentional arrays recursively.
	 */
	public static function combine($root, $rel1) {
		$arguments = func_get_args();
		return self::normalize(array_reduce($arguments, function($a,$b) {
			if(is_array($a))
				$a = array_reduce($a, 'Path::combine');
			if(is_array($b))
				$b = array_reduce($b, 'Path::combine');
 
			return "$a/$b";
		}));
	}
 
	/**
	 * Empty, private constructor, to prevent instantiation
	 */
	private function __construct() {
		// Prevents instantiation
	}
}
]]> http://www.liranuna.com/php-path-resolution-now-php-5-3-compatible/feed/ 0 Diff parser for CodeMirror http://www.liranuna.com/diff-parser-for-codemirror/ http://www.liranuna.com/diff-parser-for-codemirror/#comments Sun, 07 Feb 2010 02:56:36 +0000 LiraNuna http://www.liranuna.com/?p=1108 I wrote a quick diff parser for  CodeMirror.

CodeMirror is a real-time code editor for your browser. I know that diff isn’t a format edited by humans, but I found myself needing the diff syntax highlight where other code is shown.
More on that soon.

Live Example, Download the parser.

]]> http://www.liranuna.com/diff-parser-for-codemirror/feed/ 7 PHP Path resolution class – Relative paths made easy http://www.liranuna.com/php-path-resolution-class-relative-paths-made-easy/ http://www.liranuna.com/php-path-resolution-class-relative-paths-made-easy/#comments Sat, 05 Dec 2009 11:24:31 +0000 LiraNuna http://www.liranuna.com/?p=1079 Lately I’ve been working on a project that required me to handle a lot of file-system related operations, especially paths.

While PHP offers some basic functions to handle paths, such as basename and dirname to resolute the (direct) parent and base name of a path, it does not offer any means of normalizing or combining a path if it’s on a remote file system that is not in the server’s reach. If the files are local, it offers the function realpath.

I didn’t like the case and decided to write a ‘static’ utility class to handle file paths safely, without worrying about possible path masquerading from broken code.

I hope someone will find the result useful:

<?php
 
/**
 * @class Path
 *
 * @brief Utility class that handles file and directory pathes
 *
 * This class handles basic important operations done to file system paths.
 * It safely renders relative pathes and removes all ambiguity from a relative path.
 *
 * @author Liran Nuna
 */
final class Path
{
	/**
	 * Returns the parent path of this path.
	 * "/path/to/directory" will return "/path/to"
	 *
	 * @arg $path	The path to retrieve the parent path from
	 */
	public static function dirname($path) {
		return dirname(self::normalize($path));
	}
 
	/**
	 * Returns the last item on the path.
	 * "/path/to/directory" will return "directory"
	 *
	 * @arg $path	The path to retrieve the base from
	 */
	public static function basename($path) {
		return basename(self::normalize($path));
	}
 
	/**
	 * Normalizes the path for safe usage
	 * This function does several operations to the given path:
	 *   * Removes unnecessary slashes (///path//to/////directory////)
	 *   * Removes current directory references (/path/././to/./directory/./././)
	 *   * Renders relative pathes (/path/from/../to/somewhere/in/../../directory)
	 *
	 * @arg $path	The path to normalize
	 */
	public static function normalize($path) {
		return array_reduce(explode('/', $path), create_function('$a, $b', '
			if($a === 0)
				$a = "/";
 
			if($b === "" || $b === ".")
				return $a;
 
			if($b === "..")
				return dirname($a);
 
			return preg_replace("/\/+/", "/", "$a/$b");
		'), 0);
	}
 
	/**
	 * Combines a list of pathes to one safe path
	 *
	 * @arg $root	The path or array with values to combine into a single path
	 * @arg ...		Relative pathes to root or arrays
	 *
	 * @note		This function works with multi-dimentional arrays recursively.
	 */
	public static function combine($root, $rel1) {
		$arguments = func_get_args();
		return self::normalize(array_reduce($arguments, create_function('$a,$b', '
			if(is_array($a))
				$a = array_reduce($a, "Path::combine");
			if(is_array($b))
				$b = array_reduce($b, "Path::combine");
 
			return "$a/$b";
		')));
	}
 
	/**
	 * Empty, private constructor, to prevent instantiation
	 */
	private function __construct() {
		// Prevents instantiation
	}
}

Usage of this class is very simple, Path::basename and Path::dirname perform the same operation as PHP’s native dirname and basename, but safer:

<?php
// PHP's native basname will return '..'
echo basename('/path/to/treasure/island/monster/../..') . "\n";
// Safe basename will return 'treasure'
echo Path::basename('/path/to/treasure/island/monster/../..') . "\n";
 
// PHP's native dirname will return '/path/to/treasure/island/monster/..'
echo dirname('/path/to/treasure/island/monster/../..') . "\n";
// Safe dirname will return '/path/to'
echo Path::dirname('/path/to/treasure/island/monster/../..') . "\n";

Path::normalize will sanitize paths and return the safe real path even if it does not exist on the server:

<?php
// Normalize will 'sanitize' this path
// Result: '/path/to/candy/up/ahead/please/go/right'
echo Path::normalize(
	'///../path//to/./monster/././/' .
	'//../candy/.//./up/ahead/.//./' .
	'test//back/../..//please/go///' .
	'/left/./../right/123_test!/../'
) . "\n";

Lastly, Path::combine will combine paths from variable amount of strings and arrays to form one safe path:

<?php
// Combine paths from a relative path and root
// Result: '/var/www/www.site.com/index.html'
echo Path::combine(
	'/var/www/www.site.com/',
	'img/../css/jqueryui/../../index.html'
) . "\n";
 
// Combine will also take values from arrays
// Result: '/path/to/directory/sub/TEST/test/lastDirectory/filename.ext'
echo Path::combine(
	array(
		"/path/to",
		"folder/../directory"
	), 'sub',
	array(
		array(
			array(
				'TEST',
				'test',
			)
		),
		'lastDirectory',
	),
	'filename.ext'
) . "\n";

As always, code I post is under the WTFPL, so you can use it without any obligations.

]]> http://www.liranuna.com/php-path-resolution-class-relative-paths-made-easy/feed/ 5 libellen on devkitARM r26 http://www.liranuna.com/libellen-on-devkitarm-r26/ http://www.liranuna.com/libellen-on-devkitarm-r26/#comments Mon, 03 Aug 2009 17:30:48 +0000 LiraNuna http://www.liranuna.com/?p=1066 Thanks to a contributed patch from Iván Vodopiviz, libellen now works on the latest release of devkitARM.

libellen also recieved an official svn repository, incorporating this patch. Get libellen latest sources from svn using:

svn co http://svn.liranuna.com/libellen/trunk ellen

Current revision is 4, so this release is named libellen r4.

]]> http://www.liranuna.com/libellen-on-devkitarm-r26/feed/ 0 SSE intrinsics optimizations in popular compilers http://www.liranuna.com/sse-intrinsics-optimizations-in-popular-compilers/ http://www.liranuna.com/sse-intrinsics-optimizations-in-popular-compilers/#comments Sat, 25 Jul 2009 06:33:47 +0000 LiraNuna http://www.liranuna.com/?p=984 Lately I have been playing a lot with SSE optimizations and I really enjoy it so far – using functions to tell the compiler what instructions to use makes you feel the power in your finger tips. At first I was naive and thought the compiler will do exactly what it’s being told, assuming that you know what you’re doing – looking at the SSE intrinsic header file was mostly a bunch of calls to internal GCC functions or ‘extern’ in MSVC, suggesting that the compiler will simply follow your leadership.

I assumed wrong – the compiler will take the liberty to optimized your code even further – at points you wouldn’t even think about, though I have noticed that is not always the case with MSVC. MSVC will sometimes behave too trusting at the coder even when optimizations obviously could be made. After grasping the concept of SSE and what it could do, I quickly realized MSVC won’t optimize as good as GCC 4.x or ICC would.

I read a lot of forums about people who want to gain speed by using SSE to optimize their core math operations such as a 4D vector or a 4×4 matrix. While SSE will notably boost performance by about 10-30% depending on usage, there is no magic switch to tell the compiler to optimize your code to use SSE for you, so you need to know how to use intrinsics while actually optimizing along the way, while carefully examining the resulting assembly code.

This article will closely inspect and analyze the assembly output of 3 major compilers – GCC 4.x targeting Linux (4.3.3 in specific), the latest (stable) MSVC 2008 (Version 9.0.30729.1 SP1 in particular) and ICC 11.1.

I’ll start by declaring the options I give for each compiler – I am keeping it minimal and simple yet enough to output sane and optimized code.

GCC command line:

gcc -O2 -msse test.c -S -o test.asm

MSVC command line:

cl  /O2 /arch:SSE /c /FA test.c

ICC’s command line:

icc -O2 -msse test.c -S -o test.asm

MSVC automatically generates a file called test.asm, so no need to specify output file. Regardless of that, note the remarkable resemblance of the commands…

Basics

Let’s begin with a simple assignment test:

#include <xmmintrin.h>
 
extern void printv(__m128 m);
 
int main()
{
	__m128 m = _mm_set_ps(4, 3, 2, 1);
	__m128 z = _mm_setzero_ps();
 
	printv(m);
	printv(z);
 
	return 0;
}

This will assign m to be [1, 2, 3, 4] and z to be [0, 0, 0, 0]. Please note that the undefined “extern” function ‘printv’ is to force the compilers to not optimize out the variable and to “prove” that they are used, since we only assemble in both compilers, there is no need to actually define printv.

The variable m is actually const, but we didn’t hint to compiler. The compiler should understand ‘m’ does not change and move it into the const data section (.text). The zero vector should use the xorps opcode to generate a fast zero vector without trading off const memory (x XOR x is always 0).

The output:

MSVC:
	movss	xmm2, DWORD PTR __real@40400000 ; 3.0f
	movss	xmm3, DWORD PTR __real@40800000 ; 4.0f
	movss	xmm0, DWORD PTR __real@3f800000 ; 1.0f
	movss	xmm1, DWORD PTR __real@40000000 ; 2.0f
	unpcklps xmm0, xmm2
	unpcklps xmm1, xmm3
	unpcklps xmm0, xmm1
	call	_printv
	xorps	xmm0, xmm0
	call	_printv
 
GCC:
	movaps	.LC0, %xmm0
	call	printv
	xorps	%xmm0, %xmm0
	call	printv
 
	.LC0:
		.long	1065353216 ; 1.0f
		.long	1073741824 ; 2.0f
		.long	1077936128 ; 3.0f
		.long	1082130432 ; 4.0f
 
ICC:
	movaps    _2il0floatpacket.0, %xmm0
	call      printv
	xorps     %xmm0, %xmm0
	call      printv
 
	_2il0floatpacket.0:
		.long	0x3f800000,0x40000000,0x40400000,0x40800000

Both GCC and ICC understood that the variable ‘m’ is const, and moved it to the .text (const) section. MSVC however chose to use 4 xmm registers to create ‘m’ – it not only wrote to valuable registers that in a real application are crucial to have, it also forces the use of the stack if those registers actually contained information, which is common when inlining. It will also invalidate cache usage, since the data is in the opcode, effectively eliminating future prefetches. All compilers however, used xorps to create a zero vector, which is pleasing to see.

Arithmetic prediction

Next test is arithmetic prefiction. The test will see how the compiler deals with predefined SSE operations, such as arithmetic, much like predefined integer operations. The compiler should predict and precompute operations such as ’1+1′ and use the answer directly instead of making the CPU compute a static answer. The test is as follows:

#include <xmmintrin.h>
 
extern void printv(__m128 m);
 
int main()
{
	__m128 m = _mm_set_ps(-4, -3, -2, -1);
	__m128 one = _mm_set1_ps(1.0f);
 
	printv(_mm_and_ps(m, _mm_setzero_ps())); // Always a zero vector
	printv(_mm_or_ps(m, _mm_set1_ps(-0.0f))); // Negate all (nop, all negative)
	printv(_mm_add_ps(m, _mm_setzero_ps())); // Add 0 (nop; x+0=x)
	printv(_mm_sub_ps(m, _mm_setzero_ps())); // Substruct 0 (nop; x-0=x)
	printv(_mm_mul_ps(m, one)); // Multiply by one (nop)
	printv(_mm_div_ps(m, one)); // Division by one (nop)
 
	return 0;
}

On the first test, the compiler should always send a zero xmm register to printv, since x & 0 is always equal to 0. The rest of the tests should always result into sending the same register, since all the tests are a simple way to create a nop (no operation).

The results:

MSVC:
	movss	xmm0, DWORD PTR __real@c0800000
	movss	xmm2, DWORD PTR __real@c0400000
	movss	xmm3, DWORD PTR __real@c0000000
	movss	xmm1, DWORD PTR __real@bf800000
	unpcklps xmm3, xmm0
	xorps	xmm0, xmm0
	unpcklps xmm1, xmm2
	unpcklps xmm1, xmm3
	movaps	XMMWORD PTR tv129[esp+32], xmm0
	movaps	XMMWORD PTR _m$[esp+32], xmm1
	andps	xmm0, xmm1
	call	_printv
	movss	xmm0, DWORD PTR __real@80000000
	shufps	xmm0, xmm0, 0
	orps	xmm0, XMMWORD PTR _m$[esp+32]
	call	_printv
	movaps	xmm0, XMMWORD PTR tv129[esp+32]
	addps	xmm0, XMMWORD PTR _m$[esp+32]
	call	_printv
	movaps	xmm0, XMMWORD PTR _m$[esp+32]
	subps	xmm0, XMMWORD PTR tv129[esp+32]
	call	_printv
 
GCC:
	xorps	%xmm0, %xmm0
	call	printv
	movaps	.LC0(%rip), %xmm0
	call	printv
	movaps	.LC0(%rip), %xmm0
	call	printv
	movaps	.LC0(%rip), %xmm0
	call	printv
 
	.LC0:
		.long	3212836864
		.long	3221225472
		.long	3225419776
		.long	3229614080
 
ICC:
        xorps     %xmm0, %xmm0
        call      printv
	movaps    _2il0floatpacket.2, %xmm0
	orps      _2il0floatpacket.0, %xmm0
	call      printv
	movaps    _2il0floatpacket.0, %xmm0
	call      printv
	movaps    _2il0floatpacket.0, %xmm0
	call      printv
	movaps    _2il0floatpacket.0, %xmm0
	mulps     _2il0floatpacket.1, %xmm0
	call      printv
	movaps    _2il0floatpacket.0, %xmm0
	divps     _2il0floatpacket.1, %xmm0
	call      printv
 
	_2il0floatpacket.0:
		.long	0xbf800000,0xc0000000,0xc0400000,0xc0800000
	_2il0floatpacket.1:
		.long	0x3f800000,0x3f800000,0x3f800000,0x3f800000
	_2il0floatpacket.2:
		.long	0x80000000,0x80000000,0x80000000,0x80000000

The results are certainly interesting. MSVC has decided to not optimize the code and did exactly what it was told, resulting in redundant code. More should be noted: xorps (line 7) could’ve been moved after the unpcklps instruction (line 10) to take advantage of instruction pairing (when the processor executes the same opcode again, it’s usually faster, especially in SSE-land, where the CPU operates on large registers of 128bit). GCC’s code does exactly what we expect from a modern compiler; it performs static check for all operations. ICC seems to be selective on what it can determine, leaving out the redundant OR, multiplication and division while optimizing the others out.

Shuffles

Next test is regarding shuffles. There will be several tests regarding redundant shuffles that could be easily optimized out or merged, such as double reverses and subsequent shuffles

#include <xmmintrin.h>
 
extern void printv(__m128 m);
 
int main()
{
	__m128 m = _mm_set_ps(4, 3, 2, 1);
	m = _mm_shuffle_ps(m, m, 0xE4); // NOP - shuffles to same order
	printv(m);
 
	m = _mm_shuffle_ps(m, m, 0x1B); // Reverses the vector
	m = _mm_shuffle_ps(m, m, 0x1B); // Reverses the vector again, NOP
	printv(m);
 
	m = _mm_shuffle_ps(m, m, 0x1B); // Reverses the vector
	m = _mm_shuffle_ps(m, m, 0x1B); // Reverses the vector again, NOP
	m = _mm_shuffle_ps(m, m, 0x1B); // All should be optimized to one shuffle
	printv(m);
 
	m = _mm_shuffle_ps(m, m, 0xC9); // Those two shuffles together swap pairs
	m = _mm_shuffle_ps(m, m, 0x2D); // And could be optimized to 0x4E
	printv(m);
 
	m = _mm_shuffle_ps(m, m, 0x55); // First element
	m = _mm_shuffle_ps(m, m, 0x55); // Redundant - since all are the same
	m = _mm_shuffle_ps(m, m, 0x55); // Let's stress it again
	m = _mm_shuffle_ps(m, m, 0x55); // And one last time
	printv(m);
 
	return 0;
}

The results here should be minimum shuffles. First two tests should have no shuffle at all. Third should only have one shuffle to reverse the vector (mask = 0x1B). Forth test should merge the two shuffles into one shuffle, from mask 0xC9 and 0x2D to mask 0x4E (swap pairs). Last test should be optimized to only one shuffle, since all are ending up selecting the same value.

MSVC:
	movss	xmm1, DWORD PTR __real@40800000 ; 4.0f
	movss	xmm2, DWORD PTR __real@40400000 ; 3.0f
	movss	xmm3, DWORD PTR __real@40000000 ; 2.0f
	movss	xmm0, DWORD PTR __real@3f800000 ; 1.0f
	unpcklps xmm3, xmm1
	unpcklps xmm0, xmm2
	unpcklps xmm0, xmm3
	shufps	xmm0, xmm0, 228			; 000000e4H
	movaps	XMMWORD PTR _m$[esp+16], xmm0
	call	_printv
	movaps	xmm0, XMMWORD PTR _m$[esp+16]
	shufps	xmm0, xmm0, 27			; 0000001bH
	shufps	xmm0, xmm0, 27			; 0000001bH
	movaps	XMMWORD PTR _m$[esp+16], xmm0
	call	_printv
	movaps	xmm0, XMMWORD PTR _m$[esp+16]
	shufps	xmm0, xmm0, 27			; 0000001bH
	shufps	xmm0, xmm0, 27			; 0000001bH
	shufps	xmm0, xmm0, 27			; 0000001bH
	movaps	XMMWORD PTR _m$[esp+16], xmm0
	call	_printv
	movaps	xmm0, XMMWORD PTR _m$[esp+16]
	shufps	xmm0, xmm0, 201			; 000000c9H
	shufps	xmm0, xmm0, 45			; 0000002dH
	movaps	XMMWORD PTR _m$[esp+16], xmm0
	call	_printv
	movaps	xmm0, XMMWORD PTR _m$[esp+16]
	shufps	xmm0, xmm0, 85			; 00000055H
	shufps	xmm0, xmm0, 85			; 00000055H
	shufps	xmm0, xmm0, 85			; 00000055H
	shufps	xmm0, xmm0, 85			; 00000055H
	call	_printv
 
GCC:
	movaps	.LC0, %xmm1
	movaps	%xmm1, %xmm0
	movaps	%xmm1, -24(%ebp)
	call	printv
	movaps	-24(%ebp), %xmm1
	movaps	%xmm1, %xmm0
	call	printv
	movaps	.LC1, %xmm0
	call	printv
	movaps	.LC1, %xmm1
	shufps	$201, %xmm1, %xmm1
	shufps	$45, %xmm1, %xmm1
	movaps	%xmm1, %xmm0
	movaps	%xmm1, -24(%ebp)
	call	printv
	movaps	-24(%ebp), %xmm1
	movaps	%xmm1, %xmm0
	shufps	$85, %xmm1, %xmm0
	call	printv
 
	.LC0:
		.long	1065353216 ; 1.0f
		.long	1073741824 ; 2.0f
		.long	1077936128 ; 3.0f
		.long	1082130432 ; 4.0f
 
	.LC1:
		.long	1082130432 ; 4.0f
		.long	1077936128 ; 3.0f
		.long	1073741824 ; 2.0f
		.long	1065353216 ; 1.0f
 
ICC:
	movaps    _2il0floatpacket.0, %xmm0
	addl      $4, %esp
	shufps    $228, %xmm0, %xmm0
	movaps    %xmm0, (%esp)
	call      printv
	movaps    (%esp), %xmm0
	shufps    $27, %xmm0, %xmm0
	shufps    $27, %xmm0, %xmm0
	movaps    %xmm0, (%esp)
	call      printv
	movaps    (%esp), %xmm0
	shufps    $27, %xmm0, %xmm0
	shufps    $27, %xmm0, %xmm0
	shufps    $27, %xmm0, %xmm0
	movaps    %xmm0, (%esp)
	call      printv
	movaps    (%esp), %xmm0
	shufps    $201, %xmm0, %xmm0
	shufps    $45, %xmm0, %xmm0
	movaps    %xmm0, (%esp)
	call      printv
	movaps    (%esp), %xmm0
	shufps    $85, %xmm0, %xmm0
	shufps    $85, %xmm0, %xmm0
	shufps    $85, %xmm0, %xmm0
	shufps    $85, %xmm0, %xmm0
	call      printv
 
	_2il0floatpacket.0:
		.long	0x3f800000,0x40000000,0x40400000,0x40800000

The results are interesting and quite surprising – GCC passed all of the tests but the shuffle merge while MSVC and ICC didn’t optimize any of the shuffles. Shame. Interesting to note that GCC chose to duplicate the original reverse vector for post operations instead of caching it in an extra xmm register. (Copying registers is faster than copying memory, even if it’s aligned).

Dynamic input

All of the previous tests were about the compiler being able to make decisions about static data. Now it’s time for functions, where input and output isn’t known. A notable example of a vector operation is normalization. Here is a function to normalize an SSE vector and return a normalized copy.

__m128 normalize(__m128 m)
{
	__m128 l = _mm_mul_ps(m, m);
	l = _mm_add_ps(l, _mm_shuffle_ps(l, l, 0x4E));
	return _mm_div_ps(m, _mm_sqrt_ps(_mm_add_ps(l,
	                                   _mm_shuffle_ps(l, l, 0x11))));
}

The function is really optimized. It gives hints the compiler what should be a temporary variable and what should be reused and takes a total of 7 operations.

The results we expect are perfect projection of the SSE intrinsics to assembly using only 3 vectors (original, length and square):

MSVC:
	normalize:
		movaps	xmm2, xmm0
		mulps	xmm2, xmm0
		movaps	xmm1, xmm2
		shufps	xmm1, xmm2, 78 ; 0000004eH
		addps	xmm1, xmm2
		movaps	xmm2, xmm1
		shufps	xmm2, xmm1, 17 ; 00000011H
		addps	xmm2, xmm1
		sqrtps	xmm1, xmm2
		divps		xmm0, xmm1
		ret
 
GCC:
	normalize:
		movaps	%xmm0, %xmm2
		mulps	%xmm0, %xmm2
		movaps	%xmm2, %xmm1
		shufps	$78, %xmm2, %xmm1
		addps	%xmm2, %xmm1
		movaps	%xmm1, %xmm2
		shufps	$17, %xmm1, %xmm2
		addps	%xmm2, %xmm1
		sqrtps	%xmm1, %xmm1
		divps		%xmm1, %xmm0
		ret
 
ICC:
	normalize:
		movaps    %xmm0, %xmm3
		mulps     %xmm0, %xmm3
		movaps    %xmm3, %xmm1
		shufps    $78, %xmm3, %xmm1
		addps     %xmm1, %xmm3
		movaps    %xmm3, %xmm2
		shufps    $17, %xmm3, %xmm2
		addps     %xmm2, %xmm3
		sqrtps    %xmm3, %xmm4
		divps     %xmm4, %xmm0
		ret

Good to see that all compilers are equal in here. These are exactly the results we expected.

Inline Functions

Next test would be combining function calls and static compile-time data to get inline functions. Inline functions should embed the function’s code into the calling routine and case-optimize if possible. A classic case of inline functions is ‘abs’:

#include <xmmintrin.h>
 
extern void printv(__m128 m);
 
/* This is called _mm_abs_ps because 'abs' is a built in function
   and C does not allow overloading */
inline __m128 _mm_abs_ps(__m128 m) {
	return _mm_andnot_ps(_mm_set1_ps(-0.0f), m);
}
 
int main()
{
		// All positive
	printv(_mm_abs_ps(_mm_set_ps(1.0f, 0.0f, 0.0f, 1.0f)));
		// All negative
	printv(_mm_abs_ps(_mm_set_ps(-1.0f, -0.0f, -0.0f, -1.0f)));
		// Mixed
	printv(_mm_abs_ps(_mm_set_ps(-1.0f, -0.0f, 0.0f, 1.0f)));
}

The results we expect are perfect inlining of the function, resulting in the same vector over the three calls. A good compiler will also not duplicate the data over the three calls and reuse the same vector for the program, since the linker will most likely not do it.

MSVC:
	main:
		movss	xmm1, DWORD PTR __real@3f800000
		xorps	xmm2, xmm2
		movss	xmm0, DWORD PTR __real@80000000
		movaps	xmm3, xmm1
		movaps	xmm4, xmm2
		shufps	xmm0, xmm0, 0
		movaps	XMMWORD PTR tv166[esp+16], xmm0
		unpcklps xmm2, xmm3
		unpcklps xmm1, xmm4
		unpcklps xmm1, xmm2
		andnps	xmm0, xmm1
		call	printv
		movss	xmm2, DWORD PTR __real@bf800000
		movss	xmm1, DWORD PTR __real@80000000
		movaps	xmm0, XMMWORD PTR tv166[esp+16]
		movaps	xmm3, xmm2
		movaps	xmm4, xmm1
		unpcklps xmm1, xmm3
		unpcklps xmm2, xmm4
		unpcklps xmm2, xmm1
		andnps	xmm0, xmm2
		call	printv
		movss	xmm1, DWORD PTR __real@bf800000
		movss	xmm2, DWORD PTR __real@80000000
		xorps	xmm3, xmm3
		movss	xmm4, DWORD PTR __real@3f800000
		movaps	xmm0, XMMWORD PTR tv166[esp+16]
		unpcklps xmm3, xmm1
		unpcklps xmm4, xmm2
		unpcklps xmm4, xmm3
		andnps	xmm0, xmm4
		call	printv
 
GCC:
	main:
		movaps	.LC1, %xmm0
		call	printv
		movaps	.LC1, %xmm0
		call	printv
		movaps	.LC1, %xmm0
		call	printv
 
	.LC0:
		.long	2147483648 ; -0.0f
		.long	2147483648 ; -0.0f
		.long	2147483648 ; -0.0f
		.long	2147483648 ; -0.0f
 
	.LC1:
		.long	1065353216 ; 1.0f
		.long	0          ; 0.0f
		.long	0          ; 0,0f
		.long	1065353216 ; 1.0f
 
ICC:
	movaps    _2il0floatpacket.7, %xmm0
	addl      $4, %esp
	movaps    %xmm0, (%esp)
	andnps    _2il0floatpacket.6, %xmm0
	call      printv
	movaps    (%esp), %xmm0
	andnps    _2il0floatpacket.8, %xmm0
	call      printv
	movaps    (%esp), %xmm0
	andnps    _2il0floatpacket.9, %xmm0
	call      printv
 
	_2il0floatpacket.6:
		.long	0x3f800000,0x00000000,0x00000000,0x3f800000
	_2il0floatpacket.7:
		.long	0x80000000,0x80000000,0x80000000,0x80000000
	_2il0floatpacket.8:
		.long	0xbf800000,0x80000000,0x80000000,0xbf800000
	_2il0floatpacket.9:
		.long	0x3f800000,0x00000000,0x80000000,0xbf800000
	_2il0floatpacket.11:
		.long	0x80000000,0x80000000,0x80000000,0x80000000

This time each compiler chose it’s own way of optimizing. MSVC’s horrible assignment code in addition to it’s inability to predict static operations resulted in redundant code. ICC inlined the function, but kept some of the static data in the stack (while comfortably available on aligned read-only space) and did not perform any precomputation. GCC optimizes the code as we expected, but it “forgot” to remove the unnecessary helper vector (LC0) which is not used. This isn’t a big deal though because the linker will simply remove unreferenced const objects. GCC most likely kept it for when the inline function would have had use for it.

SSE comparison prediction

A good compiler should also predict branches and eliminate the unused code if the check is always true or false. SSE provides a way to compare 4 floats at once using the cmp*ps routines. If the result is true, the instruction puts a mask of 1s on the component. If it is false, 0. This instruction could be eliminated easily if the result is known during compile time – especially in inline functions. The test will implement the function ‘sign’ which returns 1, 0 or -1 per component.

#include <xmmintrin.h>
 
extern void printv(__m128 m);
 
inline __m128 sign(__m128 m) {
	return _mm_and_ps(_mm_or_ps(_mm_and_ps(m, _mm_set1_ps(-0.0f)),
				_mm_set1_ps(1.0f)),
			  _mm_cmpneq_ps(m, _mm_setzero_ps()));
}
 
int main()
{
	__m128 m = _mm_setr_ps(1, -2, 3, -4);
 
	printv(_mm_cmpeq_ps(m, m)); // Equal to itself
	printv(_mm_cmpgt_ps(m, _mm_setzero_ps())); // Greater than zero
	printv(_mm_cmplt_ps(m, _mm_setzero_ps())); // Less than zero
 
	printv(sign(_mm_setr_ps( 1,  2,  3,  4))); // All 1's
	printv(sign(_mm_setr_ps(-1, -2, -3, -4))); // All -1's
	printv(sign(_mm_setr_ps( 0,  0,  0,  0))); // All 0's
	printv(sign(m)); // Mixed
}

A good compiler will eliminate those checks and will create a const copy of the result, especially in places of where all comparisons fail, resulting a zero vector. In the following test, we will check the generated code of several const comparison results without sacrificing application size.

MSVC:
	movss	xmm2, DWORD PTR __real@40400000
	movss	xmm3, DWORD PTR __real@c0800000
	movss	xmm0, DWORD PTR __real@3f800000
	movss	xmm1, DWORD PTR __real@c0000000
	unpcklps xmm0, xmm2
	unpcklps xmm1, xmm3
	unpcklps xmm0, xmm1
	movaps	XMMWORD PTR _m$[esp+64], xmm0
	cmpeqps	xmm0, xmm0
	call	_printv
	xorps	xmm0, xmm0
	movaps	XMMWORD PTR tv258[esp+64], xmm0
	cmpltps	xmm0, XMMWORD PTR _m$[esp+64]
	call	_printv
	movaps	xmm0, XMMWORD PTR _m$[esp+64]
	cmpltps	xmm0, XMMWORD PTR tv258[esp+64]
	call	_printv
	movss	xmm2, DWORD PTR __real@3f800000
	movss	xmm3, DWORD PTR __real@40400000
	movss	xmm4, DWORD PTR __real@40800000
	movss	xmm0, DWORD PTR __real@40000000
	movaps	xmm1, xmm2
	shufps	xmm2, xmm2, 0
	movaps	XMMWORD PTR tv271[esp+64], xmm2
	unpcklps xmm0, xmm4
	unpcklps xmm1, xmm3
	unpcklps xmm1, xmm0
	movss	xmm0, DWORD PTR __real@80000000
	shufps	xmm0, xmm0, 0
	movaps	XMMWORD PTR tv272[esp+64], xmm0
	andps	xmm0, xmm1
	orps	xmm0, xmm2
	movaps	xmm2, XMMWORD PTR tv258[esp+64]
	cmpneqps xmm2, xmm1
	andps	xmm0, xmm2
	call	_printv
	movss	xmm2, DWORD PTR __real@c0400000
	movss	xmm3, DWORD PTR __real@c0800000
	movss	xmm1, DWORD PTR __real@bf800000
	movss	xmm0, DWORD PTR __real@c0000000
	unpcklps xmm1, xmm2
	movaps	xmm2, XMMWORD PTR tv258[esp+64]
	unpcklps xmm0, xmm3
	unpcklps xmm1, xmm0
	movaps	xmm0, XMMWORD PTR tv272[esp+64]
	andps	xmm0, xmm1
	orps	xmm0, XMMWORD PTR tv271[esp+64]
	cmpneqps xmm2, xmm1
	andps	xmm0, xmm2
	call	_printv
	xorps	xmm1, xmm1
	movaps	xmm0, xmm1
	movaps	xmm2, xmm1
	movaps	xmm3, xmm1
	unpcklps xmm1, xmm2
	movaps	xmm2, XMMWORD PTR tv258[esp+64]
	unpcklps xmm0, xmm3
	unpcklps xmm1, xmm0
	movaps	xmm0, XMMWORD PTR tv272[esp+64]
	andps	xmm0, xmm1
	orps	xmm0, XMMWORD PTR tv271[esp+64]
	cmpneqps xmm2, xmm1
	andps	xmm0, xmm2
	call	_printv
	movaps	xmm2, XMMWORD PTR _m$[esp+64]
	movaps	xmm0, XMMWORD PTR tv272[esp+64]
	movaps	xmm1, XMMWORD PTR tv258[esp+64]
	andps	xmm0, xmm2
	orps	xmm0, XMMWORD PTR tv271[esp+64]
	cmpneqps xmm1, xmm2
	andps	xmm0, xmm1
	call	_printv
 
GCC:
	movaps	.LC2(%rip), %xmm0
	movaps	%xmm0, (%rsp)
	cmpeqps	%xmm0, %xmm0
	call	printv
	xorps	%xmm0, %xmm0
	cmpltps	(%rsp), %xmm0
	call	printv
	xorps	%xmm1, %xmm1
	movaps	(%rsp), %xmm0
	cmpltps	%xmm1, %xmm0
	call	printv
	xorps	%xmm0, %xmm0
	cmpneqps	.LC3(%rip), %xmm0
	andps	.LC1(%rip), %xmm0
	call	printv
	movaps	.LC0(%rip), %xmm0
	xorps	%xmm1, %xmm1
	orps	.LC1(%rip), %xmm0
	cmpneqps	.LC4(%rip), %xmm1
	andps	%xmm1, %xmm0
	call	printv
	xorps	%xmm0, %xmm0
	cmpneqps	%xmm0, %xmm0
	andps	.LC1(%rip), %xmm0
	call	printv
	xorps	%xmm1, %xmm1
	movaps	(%rsp), %xmm0
	cmpneqps	%xmm1, %xmm0
	movaps	(%rsp), %xmm1
	andps	.LC0(%rip), %xmm1
	orps	.LC1(%rip), %xmm1
	andps	%xmm1, %xmm0
	call	printv
 
	.LC0:
		.long	2147483648
		.long	2147483648
		.long	2147483648
		.long	2147483648
	.LC1:
		.long	1065353216
		.long	1065353216
		.long	1065353216
		.long	1065353216
	.LC2:
		.long	1065353216
		.long	3221225472
		.long	1077936128
		.long	3229614080
	.LC3:
		.long	1065353216
		.long	1073741824
		.long	1077936128
		.long	1082130432
	.LC4:
		.long	3212836864
		.long	3221225472
		.long	3225419776
		.long	3229614080
 
ICC:
	movaps    _2il0floatpacket.8, %xmm0
	addl      $4, %esp
	cmpeqps   %xmm0, %xmm0
	call      printv
	xorps     %xmm0, %xmm0
	cmpltps   _2il0floatpacket.8, %xmm0
	call      printv
	movaps    _2il0floatpacket.8, %xmm0
	xorps     %xmm1, %xmm1
	cmpltps   %xmm1, %xmm0
	call      printv
	movaps    _2il0floatpacket.9, %xmm0
	movaps    _2il0floatpacket.9, %xmm2
	andps     _2il0floatpacket.10, %xmm0
	xorps     %xmm1, %xmm1
	cmpneqps  %xmm1, %xmm2
	orps      _2il0floatpacket.11, %xmm0
	andps     %xmm2, %xmm0
	call      printv
	movaps    _2il0floatpacket.12, %xmm1
	movaps    _2il0floatpacket.10, %xmm0
	andps     %xmm1, %xmm0
	orps      _2il0floatpacket.11, %xmm0
	xorps     %xmm2, %xmm2
	cmpneqps  %xmm2, %xmm1
	andps     %xmm1, %xmm0
	call      printv
	xorps     %xmm0, %xmm0
	cmpneqps  %xmm0, %xmm0
	call      printv
	movaps    _2il0floatpacket.10, %xmm0
	movaps    _2il0floatpacket.8, %xmm1
	andps     %xmm1, %xmm0
	orps      _2il0floatpacket.11, %xmm0
	xorps     %xmm2, %xmm2
	cmpneqps  %xmm2, %xmm1
	andps     %xmm1, %xmm0
	call      printv
 
	_2il0floatpacket.8:
		.long	0x3f800000,0xc0000000,0x40400000,0xc0800000
	_2il0floatpacket.9:
		.long	0x3f800000,0x40000000,0x40400000,0x40800000
	_2il0floatpacket.10:
		.long	0x80000000,0x80000000,0x80000000,0x80000000
	_2il0floatpacket.11:
		.long	0x3f800000,0x3f800000,0x3f800000,0x3f800000
	_2il0floatpacket.12:
		.long	0xbf800000,0xc0000000,0xc0400000,0xc0800000
	_2il0floatpacket.14:
		.long	0x80000000,0x80000000,0x80000000,0x80000000
	_2il0floatpacket.15:
		.long	0x3f800000,0x3f800000,0x3f800000,0x3f800000

None of the compilers optimized the comparisons, which could benefit the code in a large extent, especially when inlined. It’s notable to mention that GCC merged some of constants, eliminating 2 of the vectors that ICC left. ICC and GCC both optimized useless ORs where possible while MSVC simply followed the code intrinsic by intrinsic.

Conclusion

I keep hearing the catch-phrase among programmers that “the compiler is better than you [think].” I completely disagree with it and object the use of it. Not only it makes novice programmers misunderstand it and give the compiler a lot of credit where it’s impossible to expect a compiler to optimize a case, it also makes more advance programmers become lazy and believe the compiler does know what it’s doing.

Proven here is a case using the so called ‘intrinsics’ to guide the compiler as opposed of instructing it. As seen by the above examples, only GCC (and to an extent, ICC) behaves the way we expect it to though it still misses a few of the cases (such as merging shuffles and predicting vector branches). MSVC is most likely the worst example of an SSE-guided compiler – not only it did not optimize any of the tests, it generated horrible assignment code which abused the stack most of the time and hurt performance by not utilizing cache properly.

If you are to code using SSE intrinsics, I advise you to take a closer look at the code if you want maximum performance. Taking advantage of SSE for speed will result a lot of satisfaction if used properly – instruction pairing, redundant arithmetic operations and redundant compares should be optimized by human beings most of the time and you should not rely on the compiler to do that. Compilers are given much more credit than they deserve.

As a side note about GCC’s near perfection in code generation – I was quite surprised seeing it surpass even Intel’s own compiler! It shows that even compiler writers, who know their own hardware and internal mechanisms, can overlook simple problems in the way humans think – redundancy in most cases. I highly recommend giving the newest GCC 4.4 a try, if you are on Linux, you most likely have GCC 4.3.x, or if your distribution is an early bird (Gentoo, Fedora…), you might already have it. Windows users are lucky enough to know that GCC 4.4 have been ported successfully to Windows on both the MinGW suite and the TDM suite. Mac users might have to compile gcc 4.4 themselves using Xcode (which is actually gcc 4.0.1).

Happy optimizing!

]]> http://www.liranuna.com/sse-intrinsics-optimizations-in-popular-compilers/feed/ 10 Securing your Debian server against slowloris http://www.liranuna.com/securing-your-debian-server-against-slowloris/ http://www.liranuna.com/securing-your-debian-server-against-slowloris/#comments Sat, 27 Jun 2009 02:26:38 +0000 LiraNuna http://www.liranuna.com/?p=949 I recently came across a very nasty DoS attack that any script kiddy can execute – called slowloris. It involves sending partial HTTP packets while Apache2 patiently waits for an unreasonable amount of time for the remaining data while consuming a thread, doing so continuously will prevent Apache2 from opening more threads and serving potential web viewers.

One old remedy for this was supposedly mod_evasive, but it doesn’t really work against that specific type of attack as it acts too late to understand it’s an attack.

Very recently, an Apache mod fixing this vulnerability had been released – mod_antiloris, but it’s made with a RedHat based server in mind. Here are the steps to get it working on a Debian or any other Debian compatible server (such as Ubuntu).

First install the prerequisites. I assume you are using the threaded version of Apache, else you are not vulnerable to this type of attack.

sudo apt-get install gcc apache2-threaded-dev

Next, get the module source, extract it and compile:

wget "ftp://ftp.monshouwer.eu/pub/linux/mod_antiloris/mod_antiloris-0.3.tar.bz2"
tar xvf mod_antiloris-0.3.tar.bz2
cd mod_antiloris-0.3/

The following command will end up in error – this is perfectly normal! Since apxs2 (Apache extension service) for Debian isn’t modified to handle Debian-style modules, do not run it as root as it will mess up with your system, thinking it’s RedHat compatible.

apxs2 -a -i -c mod_antiloris.c

Because apxs2 didn’t have permission to copy the module, we’ll do it ourselves:

sudo cp .libs/mod_antiloris.so /usr/lib/apache2/modules/mod_antiloris.so

Now we’ll add Debian-style .load file to auto load the module:

sudo su -c "echo 'LoadModule antiloris_module /usr/lib/apache2/modules/mod_antiloris.so' > /etc/apache2/mods-available/antiloris.load"

Then we’ll enable the module, Debian style:

sudo a2enmod antiloris

And reload Apache’s configurations and modules:

sudo /etc/init.d/apache2 reload

This module solves the slowloris DoS attack – so I urge you to install it as soon as possible if you are using Apache as your HTTP server.

I would like to make sure credit is where it is due – I did not develop this module – I just wrote instructions on how to make it Debian compatible, since it seems to be RedHat centric. The module was written and hosted by Kees Monshouwer, which I cannot seem to find any official website associated with.
I hope this will help people as much as it helped me.

]]> http://www.liranuna.com/securing-your-debian-server-against-slowloris/feed/ 8 Sintia has a Twitter page http://www.liranuna.com/sintia-has-a-twitter-page/ http://www.liranuna.com/sintia-has-a-twitter-page/#comments Sat, 13 Jun 2009 08:09:12 +0000 LiraNuna http://www.liranuna.com/?p=947 Sintia, the suicidal MegaHAL IRC service bot I run for over 6 years on EFnet now have a twitter page! Sintia will be updating it regularly.

Check Sintia’s page here.

]]> http://www.liranuna.com/sintia-has-a-twitter-page/feed/ 1 Lessons updated http://www.liranuna.com/lessons-updated/ http://www.liranuna.com/lessons-updated/#comments Fri, 01 May 2009 02:47:37 +0000 LiraNuna http://www.liranuna.com/?p=876 My aging NDS 2D lessons have been updated to conform to the latest version of libnds. They should now compile and run on a modern devkitARM toolchain.

I hope someone will find this useful.

]]> http://www.liranuna.com/lessons-updated/feed/ 4 NDS Blending Demo http://www.liranuna.com/nds-blending-demo/ http://www.liranuna.com/nds-blending-demo/#comments Sun, 05 Apr 2009 02:16:09 +0000 LiraNuna http://www.liranuna.com/?p=858 blending-demo

First I would like to start with the fact that this demo was lying around in my HDD since Halloween.

This demo was written to demonstrate how easy it is to utilize the DS’s hardware blending and create impressive effects with no effort. In this demo, the witch is flying in the sky, and whenever she’s hovering between the moon, she turns black, because the light from the moon illusions it as such.

The demo is composed of 2 backgrounds and a sprite. the sprite is set to blend with the first background (the moon) with 0 blending, resulting the black color.

Download: blending-demo.

]]> http://www.liranuna.com/nds-blending-demo/feed/ 1 Rhythmbox ‘Now Playing’ script for XChat http://www.liranuna.com/rhythmbox-now-playing-script-for-xchat/ http://www.liranuna.com/rhythmbox-now-playing-script-for-xchat/#comments Sat, 28 Mar 2009 04:39:58 +0000 LiraNuna http://www.liranuna.com/?p=846 This is a small script I wrote for myself when using XChat and Rhythmbox.

Ever since the latest Rhythmbox release, there has been an undocumented feature in rhythmbox-client to print the string received from shoutcast streams, such as my favorite di.fm radio, which I normally have on. I found several xchat-rhythmbox announcers but they all lacked the ability to determine if rhythmbox currently streams music or listens to a music file.

Now that I actually have free time, I could write a small script to do exactly what I wanted, and I’ve decided to share it. The source/script is released under the terms of the WTFPL.

Download link: rhythmbox_nowplaying.tar.gz.

]]> http://www.liranuna.com/rhythmbox-now-playing-script-for-xchat/feed/ 10