Artur Hefczyc

This is really annoying. Why something has to be simple if it can be complicated?

For some reason on Mac OS X commands: 'ulimit' and 'sysctl' works fine but are actually useless. Don't count on them if you want to change maximum number of open files in your system. Fortunately this forum threads comes to help. To save you from reading through all of this here is how it goes:

To check limits on your system run: 'launchctl limit'. To change max number of open files run: 'launchctl limit maxfiles 400000 unlimited'. To change the setting permanently add to the file: '/etc/launchd.conf' following line:

limit maxfiles  400000           unlimited

This doesn't however work for maxproc setting and please don't ask my why. I am asking...

Problem description

Let's say we have N queues with packets waiting for processing. Queues are indexed from 0..N-1. Queue no 0 is the highest priority. Queue no N-1 is the lowest priority. Packets are being added to all queues asynchronously all the time and there is one thread reading packets from the queues.

Question

How to efficiently empty all the queues in a single thread and make sure packets from the highest priority queues are always processed as soon as possible.

Starting point

// Queues are kept in a map as below.
// EnumMap<Priority, LinkedBlockingQueue<Packet>> queues
while (! stopped) {
  try {
    // Your code starts here:
    ................
    // Your code ends here.
    // Let's make sure there was nothing added to any queue in the meantime
    synchronized (queues) {
      boolean added = false;
      for (LinkedBlockingQueue<Packet> tmpqueue : queues.values()) {
        if (added = (tmpqueue.size() > 0)) {
          break;
        }
      }
      if (!added) {
        queues.wait();
      }
    }
  } catch (Exception e) {
    log.log(Level.SEVERE, "[" + getName() +
      "] Exception during packet processing: " + packet.toString(), e);
  } // end of try-catch
} // end of while (! stopped)

Any suggestions are very welcomed.

 Have a look at this article about increased DDOS attacks on the services driven by the bot nets. In fact bot nets drive attacks but the real vehicles they drive are millions of Windows machines over the Internet.

Linux might be not the friendlies operating system yet. There are problems with hardware yet which make it difficult to use sometimes. But maybe defending yourselves and others from intruders is worth the extra effort. On the other side I am sure the more of us use the Linux the more hardware vendors add Linux drivers for their hardware.

Sometimes, when you change the Emacs version and you have own or custom Emacs packages installed and compiled with the previous version you may get unexpected behavior, including invalid function c-point warning message. The best solution for this is recompiling your *.el files using the current Emacs version:

Remove all old *.elc pre-compiled files:

find site-lisp/ -name "*.elc" -print | xargs rm -f

Recompile all Emacs source code files:

emacs --batch --eval "(byte-recompile-directory \"site-lisp/\" 0)"

The simple bash loop to do something with all files in the directory is:

for f in * ; do echo $f ; done

which just displays all files in this case.

This is easy, what I usually need and often forget how to do it is to execute some action exact number of times. Something like simple for loop in C or Java. And there is an exact match for such loop in Bash too:

for (( i=1 ; i<=100 ; ++i )) ; do echo $i ; done 

I found another similar solution on the Internet today at spiralbound.net:

for i in $(seq 1 100); do echo $i ; done

which might useful in some cases...

 I used to use KMail for years. It had some issues especially with IMAP accounts but worked fine for the most of the time and problems could be easily avoided by careful usage.

The Mail however in Mac OS X is completely different story. It is quite stable and better at using the IMAP accounts but it is consistently annoying in certain places.

Well, the installation went smooth and without any problems. I have made one mistake though. For some unknown to me reason I selected Mac UK keyboard as one of final settings. After restart there was no way to login or change the  settings. Now reinstalling Gentoo again.

Very short guide for those who are interested how to install Gentoo in less then 30 minutes:

1. Get the livecd-i686-installer-2007.0.iso or more recent image if it is available. Just make sure you download the full live CD not the minimal version.
2. Setup the virtual machine in VMWare and point it to the ISO image as an installation disk. After you finished setting it up VMWare tries to run it automatically. Don't let it to do so.
   • Depending on your needs set proper HDD size. It doesn't need to be very huge as you can always attach another virtual HDD later on. On the other hand HDD in VMWare starts at zero size and grows when it is needed up to the size you set. So in theory you can set quite huge size. I set 20GB for my installation.
   • I would recommend a least 1GB RAM. But again it depends what your are going to do with the Gentoo installed.
3. Stop it and go to settings and at least adjust memory settings. Defaults are usually quite low. I set 1.5GB for the Gentoo as my Macbook has 4GB.
4. Now, after you adjusted your all settings (in my case VMWare always crashes at least once) start virtual machine and let it boot from the ISO image.
5. The best and quickest way to install it is to use Networkless installation mode provided by the GUI installer. But before you run it I would recommend to setup partitions on your hard drive manually as the defaults offered by the installer are not the best for my preferences. My preferred layout (for private desktop, workstation) is: 100MB /boot partition, 2GB swap and the rest of hard drive for / - the rest of the system including home and var directory. On the server I usually set a separate partition for /home. Here is a step by step guide how you set partitions for your installation:
   • Browser menu: Applications -> Accessories and click Terminal
   • Terminal opens so set the root password to whatever you like using command: sudo passwd
   • When the new password is set switch to root user with command: su - and when asked enter the password you just set
   • Set your hard drive partitions with fdisk: fdisk /dev/sda
     • 'n' - creates a new partition, when asked what type of the partition to create answer 'p' - primary, and then press '1' - the first partition. It asks you for the first sector number, press Enter to accept default. It asks you then for the last sector number or the size of the partition, enter: +100M - 100 MB size as this is your boot partition.
     • 'n' - creates next partition, when asked about partition type answer 'p' (primary) again and press '2' - the second partition. Accept the default first sector and enter +2000M (2GB swap) as the size (last sector).
     • 'n' - creates your third and last partition, again 'p' for primary partition and '3' - the third partition. Accept the default first and if you accept the default for the last sector the rest of the whole hard disk will be allocated.
     • Change the second partition type from the default Linux to Linux swap by pressing 't' and then select the second partition by pressing '2' and when asked for the Hex code enter 82.
     • You can now press 'p' to display your partition table to make sure everything is set as it is expected.
     • 'w' to write your new partition table and exit fdisk.
   • Now you have to create filesystems on your new partitions:
     • mke2fs /dev/sda1 - creates ext2 filesystem on your /boot partition
     • mkswap /dev/sda2 - creates swap on your second, swap partition
     • mkreiserfs /dev/sda3 - creates Reiserfs file system on your third / the system partition

Now you can run Gentoo Linux Installer (GTK+) and if you select Networkless installation you have Gentoo system setup in less then half an hour.

Well, I am not really sure whether I install Linux on Macbook as a primary OS. However I miss the days when I used Linux as my development platform and there are some stuff in Mac OS X I just can't get used to.

So the day has come and I am installing Gentoo inside VMWare Fusion. The Macbook Pro has enough power to run both at the same time and with 4GB RAM on board there is enough for both systems. I hope this was I can use benefits from both worlds without sacrificing anything. On top of that I normally use 2 screens - one built-in the laptop and the second on the external monitor. This way I can use both system in full screen mode.

I tried both VM available for Mac OS X: Parallels and VMWare. I wanted to install amd64 Gentoo version. Parallels doesn't support 64bit systems at all so it lost at the beginning and it hang a few time so hard, I couldn't even kill -9 it. Well, this is something for OS to blaim for... VMWare claims it does. It even boots system from CD. During installation however it hangs too. The CD is not broken, nor the installation program is as www.tigase.org has been installed from this CD.

The Gentoo is installing right now in VMWare, let see how it goes...

I am generally happy with the Mac OS X installed on my Macbook. It is quite a good system and in most cases it works as expected.

Sometimes however it drives my crazy. I don't think this is a system created for software developers. At least not for software developers who work on the networking software.

I tried to do final tests over my Tigase Jabber/XMPP Server server and just couldn't do what I needed. One of the tests, called performance test, require sending 1000 messages over concurrent 1000 network connections between the Testsuite and the Tigase server.  Whatever I did, I couldn't successfully open more than 900 concurrent connections. I tried to play with numerous setting, like maximum opened files, file descriptors, tweaking TCP/IP settings, nothing helped.

On Linux I can easily manage to open 60k connections and all works fine without failure. I wonder whether this is kind of a hidden configuration parameter or maybe it is a desktop version of the Mac OS X feature and to open more connections I have to have server version...

Any suggestion or help with forcing Mac OS X to handle more network connections would be very much appreciated. Otherwise I will probably replace the nice system with the system which works...

I always wondered if the multi-core or hyper-threading technology is only marketing thing or ot really gives you a significant performance boost in your system. I mean home based system not in enterprise servers. Are all those core really used and given that you have a shared memory and a shared hard drive can you make use of all those cores.

I had to replace my old server which hosts all tigase.org services (websites, e-mails, Jabber/XMPP, SVN, project tracking....) as it couldn't cope with the load and it had too little memory. And as usually upgrading memory in old systems is much more expensive that buying memory for the new machine.

At the end I decided to build a new server from scratch. 64bits, 4GB RAM, 500GB HDD in Raid-1 and a CPU. I wasn't really convinced for multi-core so I thought for a moment about buying multi-cpu motherboard. I coulnd't, however find any non-Xeon multi CPU motherboards and XEON motherboards were quite expensive, don't even mention about CPUs.

Ok, I turned back to multi-core. Quad core was not so much more than Dual then I gave it a go.

Gentoo installation was quite smooth and suprisingly fast. Even though I've put -j5 option just in case to the make command didn't believe too much in it.

The compilation was so quick so I wanted to see what is the real gain from the multicore system. The best test as usual is the kernel compilation. I ran compilation with a single thread and then two threads, three, four, five and six. Below are results:

 Hm, colcusion: Number of CPU cores does really matter. Even in home based systems...