How to Spot & Solving IO Bottleneck problem in Vmware Server 2.x with SSD HDD

In this article, I will try to point my solution for solving IO Bottleneck in Vmware Server 2.x

Firstly, why use virtual solution in the first place? Well there are many subjective reasons, and but for me my main reason is Electrical Consumption. Running multiple guest OS under the same physical Server will greatly reduce electrical consumption. At the moment, our server ( AMD X6 with 8GB RAM) are hosting 10 Guest OS : 3 pfsense, 5 ubuntu server, 1 IPCOP, 1 Windows XP. I can save up to 90 %.

Some of my friend ask me, there is gonna be performance hit by running virtualization. Of course there is, but, in real world application, it’s very rare to see your server running in 100% utilization, since most of the time, it’s doing idle/nothing. Performance hits can be spotted when it start to slow down / under perform, most of it are caused by bottleneck in CPU/Memory/IO or hardware problem.

How to Spot Bottleneck :

If you run vmware server under linux, you can use TOP command to monitor server load. Here is an example:

root@vmserver002:~# top
top – 11:06:56 up 5 days, 8 min, 2 users, load average: 1.42, 1.75, 1.70
Tasks: 244 total, 1 running, 243 sleeping, 0 stopped, 0 zombie
Cpu(s): 2.9%us, 15.7%sy, 0.0%ni, 80.6%id, 0.3%wa, 0.1%hi, 0.4%si, 0.0%st

Mem: 8193488k total, 8118824k used, 74664k free, 38152k buffers
Swap: 0k total, 0k used, 0k free, 6919920k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

3995 root 20 0 1270m 56m 39m S 54 0.7 1772:31 vmware-vmx
3427 root 20 0 697m 251m 236m S 21 3.1 1167:04 vmware-vmx

3434 root 20 0 559m 66m 52m S 17 0.8 1739:46 vmware-vmx
4726 root 20 0 697m 205m 191m S 13 2.6 763:06.36 vmware-vmx

3443 root 20 0 707m 140m 118m S 8 1.8 304:12.98 vmware-vmx
3927 root 20 0 972m 63m 48m S 8 0.8 205:20.49 vmware-vmx

3957 root 20 0 558m 18m 6860 S 8 0.2 654:20.00 vmware-vmx
4734 root 20 0 843m 342m 320m S 8 4.3 267:09.66 vmware-vmx

4626 root 20 0 432m 129m 106m S 6 1.6 218:49.46 vmware-vmx
3492 root 20 0 706m 83m 60m S 4 1.0 184:30.86 vmware-vmx

3284 root 20 0 139m 54m 12m S 2 0.7 53:59.05 vmware-hostd
3440 root 20 0 0 0 0 S 2 0.0 62:27.59 vmware-rtc

As you can see, I run 10 guest OS, and the CPU load are +- 20% (80% are idle). And this is a production server, not testing server.

Cpu(s): 2.9%us, 15.7%sy, 0.0%ni, 80.6%id, 0.3%wa, 0.1%hi, 0.4%si, 0.0%st

legend:

us : % CPU used for user space application
sy : % CPU used for system space application
ni : % CPU used for application with nice attribute
id : % CPU not used / IDLE
wa : % CPU in waiting due to bottleneck in IO

Pay attention to the "wa" attribute. When I use regular IDE/SATA HDD, the "%wa" is most of the time reach double digit and the total of "%us" + "%sy" is hardly ever reach 10%, that means, my server is under IO bottleneck. Ever since I switch to SSD HDD, which are pretty cheap nowadays, "%wa" is always very low, just around 4-5% at peak load (at system boot or when starting guest OS simultaneously.

So, upgrade your hard drive to SSD. Your system may reside at the old IDE/SATA hard disk, but make sure the storage for guest OS are reside in the SSD HDD. Also consider investing in main board with SATA 3 / SATA 6 port. My main board is still SATA 3, with both the main HDD and SSD HDD are in the same controller, and it’s not a real issue.

Grup Diskusi KPO BMD Terban resmi dilaunch

Silakan bapak ibu orang tua murid BMD Terban untuk berkunjung ke grup diskusi KPO BMD Terban di Googlegroups :

https://groups.google.com/forum/#!forum/kpo-budi-mudia-dua-terban

DIY : CCTV Server with Ubuntu Server 12.04 + Motion + USB Webcam(s)

Building CCTV Server with Ubuntu Server 12.04 + Motion + USB Webcam(s)

What you need:

1. PC with Ubuntu Server Installed
2. PCI USB Card(s), as 1 USB Controller only capable of handling 2 USB Webcams
3. Webcam(s). Make sure they work in linux. I have bought cheap webcams from Ebay, which are detected automatically by Ubuntu
4. Optional : IR Illuminator
5. WIP :  Utilizing USB RJ45 Extender

Steps:

1. Preparing Server & installing Motion

Update your server repo to the latest :

 sudo apt-get update

Install motion :

sudo apt-get install motion

2. Tweaking uvcvideo driver

sudo rmmod uvcvideo
sudo modprobe uvcvideo quirks=128

This will be reset every reboot. If this works, create the following file:

sudo pico /etc/modprobe.d/uvcvideo.conf

containing the line:

options uvcvideo quirks=128

3. Configuring Motion for multiple threads, storage location, etc

The configuration file is located at : /etc/motion/motion.conf

Suggestion : comment out the device in motion conf and configure each camera with their separate thread conf file

Make sure to add line of each thread file in the last line of the motion.conf

thread /etc/motion/camera1.conf

thread /etc/motion/camera2.conf

…..

thread /etc/motion/camera6.conf

Example content of camera1.conf ( pay attention to bold line):

root@cctv01:~# cat /etc/motion/camera1.conf
# /usr/local/etc/thread1.conf
#
# This config file was generated by motion 3.2.12

 

###########################################################
# Capture device options
############################################################

# Videodevice to be used for capturing (default /dev/video0)
# for FreeBSD default is /dev/bktr0
videodevice /dev/video2

# The video input to be used (default: 8)
# Should normally be set to 1 for video/TV cards, and 8 for USB cameras
input 8

# Draw a user defined text on the images using same options as C function strftime(3)
# Default: Not defined = no text
# Text is placed in lower left corner
text_left CAMERA 1
############################################################
# Target Directories and filenames For Images And Films
# For the options snapshot_, jpeg_, mpeg_ and timelapse_filename
# you can use conversion specifiers
# %Y = year, %m = month, %d = date,
# %H = hour, %M = minute, %S = second,
# %v = event, %q = frame number, %t = thread (camera) number,
# %D = changed pixels, %N = noise level,
# %i and %J = width and height of motion area,
# %K and %L = X and Y coordinates of motion center
# %C = value defined by text_event
# Quotation marks round string are allowed.
############################################################

# Target base directory for pictures and films
# Recommended to use absolute patch. (Default: current working directory)
#target_dir /usr/local/apache2/htdocs/cam1
############################################################
# Live Webcam Server
############################################################

# The mini-http server listens to this port for requests (default: 0 = disabled)
webcam_port 8081

# Command to be executed when a picture (.ppm|.jpg) is saved (default: none)
# The filename of the picture is appended as an argument for the command.
#on_picture_save /usr/local/motion-extras/camparse1.pl

# Command to be executed when a movie file (.mpg|.avi) is closed. (default: none)
# Filename of movie is appended as an argument for the command.
#on_movie_end /usr/local/motion-extras/mpegparse1.pl

Test & run motion :

sudo /etc/init.d/motion start

check syslog, make sure no error. You should see something like this:

Jul 20 10:43:52 cctv01 motion: [5] Using palette YUYV (352×288) bytesperlines 704 sizeimage 202752 colorspace 00000000
Jul 20 10:43:52 cctv01 motion: [5] found control 0×00980900, “Brightness”, range 0,255
Jul 20 10:43:52 cctv01 motion: [5] #011″Brightness”, default 140, current 102
Jul 20 10:43:52 cctv01 motion: [5] found control 0×00980901, “Contrast”, range 0,127
Jul 20 10:43:52 cctv01 motion: [5] #011″Contrast”, default 36, current 36
Jul 20 10:43:52 cctv01 motion: [5] found control 0×00980902, “Saturation”, range 0,127
Jul 20 10:43:52 cctv01 motion: [5] #011″Saturation”, default 80, current 80
Jul 20 10:43:52 cctv01 motion: [5] found control 0×00980903, “Hue”, range -15,15
Jul 20 10:43:52 cctv01 motion: [5] #011″Hue”, default -3, current -3
Jul 20 10:43:52 cctv01 motion: [5] found control 0×00980910, “Gamma”, range 1,3
Jul 20 10:43:52 cctv01 motion: [5] #011″Gamma”, default 2, current 2
Jul 20 10:43:52 cctv01 motion: [5] mmap information:
Jul 20 10:43:52 cctv01 motion: [5] frames=4
Jul 20 10:43:52 cctv01 motion: [5] 0 length=202752
Jul 20 10:43:52 cctv01 motion: [5] 1 length=202752
Jul 20 10:43:52 cctv01 motion: [5] 2 length=202752
Jul 20 10:43:52 cctv01 motion: [5] 3 length=202752
Jul 20 10:43:52 cctv01 motion: [5] Resizing pre_capture buffer to 1 items
Jul 20 10:43:53 cctv01 motion: [2] Started stream webcam server in port 8082
Jul 20 10:43:53 cctv01 motion: [2] Resizing pre_capture buffer to 2 items
Jul 20 10:43:53 cctv01 motion: [6] Started stream webcam server in port 8086
Jul 20 10:43:53 cctv01 motion: [6] Resizing pre_capture buffer to 2 items
Jul 20 10:43:53 cctv01 motion: [1] Started stream webcam server in port 8081
Jul 20 10:43:53 cctv01 motion: [1] Resizing pre_capture buffer to 2 items
Jul 20 10:43:53 cctv01 motion: [5] Started stream webcam server in port 8085
Jul 20 10:43:53 cctv01 motion: [5] Resizing pre_capture buffer to 2 items
Jul 20 10:43:53 cctv01 motion: [4] Started stream webcam server in port 8084
Jul 20 10:43:53 cctv01 motion: [4] Resizing pre_capture buffer to 2 items
Jul 20 10:43:53 cctv01 motion: [3] Started stream webcam server in port 8083
Jul 20 10:43:53 cctv01 motion: [3] Resizing pre_capture buffer to 2 items

4. IR Filter Mod &  utilizing IR Illuminator

http://www.instructables.com/id/Hacked-WebCam-for-IR-Photography-and-Night-Vision/

5. Mod for outdoor usage:

http://www.instructables.com/id/Convert-a-Webcam-to-work-outdors-in-bright-light/

6. BROKEN WIP : Utilizing USB RJ45 Extender

useful links :

http://renoirsrants.blogspot.com/2011/07/multiple-webcams-on-zoneminder.html

http://adhi.widjajanto.net/blog/archives/2011/12/my_home_server.html (in bahasa Indonesia)

http://www.instructables.com/id/Hacked-WebCam-for-IR-Photography-and-Night-Vision/

http://www.instructables.com/id/Convert-a-Webcam-to-work-outdors-in-bright-light/

Percobaan upload file

listrik april 2012

pfsense 2 under VMware server 2( was Work on progress : running pfsense 2.0 (based on freebsd 8.1) under xenserver 5)

Few months ago, I had zero success with pfsense 2.0 under xenserver. This project is pretty important, since pfsense 2.0 offers a lot of improvement.

Posted with WordPress for BlackBerry.

Update March 2011:

A lot of thing has changed. Xenserver has finally given its place for VMware server 2. One of the reason is the later ability to host pfsense 2 guest.

So far with RC version, everything is running pretty nice. I love the traffic shape – limiter feature, useful to limit client connection with multiple combination, source/destination/total bucket based rules. Thanks pfsense, and VMware.

Room for improvement/ more homework:
1. Managing optimal guest bootup delay at startup.
2. Tweaking network performance.

Stay tuned.

Clevo TN120R Touchscreen TC4UM WORKED in Ubuntu Jaunty & Karmic

This how to probably will only work for ‘ET&T Technology TC4UM with device  ID 0664:0306 on Clevo TN120R running Ubuntu Hardy/Intrepid/Jaunty/Karmic.

• STEP 1: Download the driver from Magictouch website
• STEP 2: Copy Modules and other files

Extract the driver from magictouch website (Linux_USB_V2.20_ub904_090430[i386].tar.gz) and you will find these objects:

drwxr-xr-x 4 dedieko dedieko 4096 2009-04-29 10:19 .
drwxr-xr-x 3 dedieko dedieko 4096 2009-11-26 10:35 ..
drwx—— 2 dedieko dedieko 4096 2009-11-26 07:16 driver
-rwxrwxrwx 1 dedieko dedieko 4116 2009-04-28 14:18 install_Ubuntu9.04
-rwxrwxrwx 1 dedieko dedieko 2085 2009-04-28 14:28 uninstall_Ubuntu9.04
drwx—— 2 dedieko dedieko 4096 2009-11-26 07:16 utility

Instead of running Install_Ubuntu9.04, I wanted to know how things work, so we go with the manual way.
Go to driver folder and you will find 1 file:

-rwxr-xr-x 1 dedieko dedieko 75285 2009-04-29 10:05 etouch_drv.so

Copy this file to /usr/lib/xorg/modules/input
Go to utility folder and youwill find 3 files:

-rwxr-xr-x 1 dedieko dedieko 31706 2009-04-29 09:12 Calibration
-rwxr-xr-x 1 dedieko dedieko 14829 2008-12-10 14:33 Lift-off
-rwxr-xr-x 1 dedieko dedieko 29058 2009-04-28 15:57 Swap

Copy these files to /etc/etandt folder. Don’t forget to change the file ownership. Safest bet is to chmod 777

• STEP 3: Configure xorg.conf

If you use freshly installed Karmic, there is no xorg.conf, you could run this command to generate default Xorg.conf:

sudo dpkg-reconfigure -phigh xserver-xorg

Then edit the newly generated xorg.conf. Add ServerLayout Section and InputDevice Section
The hardest thing is probably to pinpoint the device location. Type this:

lshal | grep hiddev

Mine is look like this:

dedieko@dedieko-laptop:~$ lshal | grep hiddev
udi = ‘/org/freedesktop/Hal/devices/usb_device_664_306_noserial_if0_hiddev’
  hiddev.application_pages = {‘Unknown page 0xd0004′} (string list)
  hiddev.device = ‘/dev/usb/hiddev0‘  (string)
  hiddev.product = ‘ET&T Technology TC4UM’  (string)
  info.capabilities = {‘hiddev’} (string list)
  info.category = ‘hiddev’  (string)
  info.udi = ‘/org/freedesktop/Hal/devices/usb_device_664_306_noserial_if0_hiddev’  (string)
  linux.device_file = ‘/dev/usb/hiddev0′  (string)
  linux.sysfs_path = ‘/sys/devices/pci0000:00/0000:00:1d.2/usb7/7-1/7-1:1.0/usb/hiddev0′  (string)
dedieko@dedieko-laptop:~$

Pay attention to the red line, now write that or memorize it, then put it in the Xorg.conf below.
Mine is look like this:

Section “Monitor”
    Identifier    “Configured Monitor”
EndSection

Section “Screen”
    Identifier    “Default Screen”
    Monitor        “Configured Monitor”
    Device        “Configured Video Device”
    SubSection “Display”
        #Virtual    2560 1024
        Virtual    4096 4096
    EndSubSection
EndSection

Section “Device”
    Identifier    “Configured Video Device”
    Option “AccelMethod” “UXA”
    #Option “MigrationHeuristic” “Greedy”
EndSection

Section “ServerFlags”
    Option    “DontZap”    “False”
EndSection

Section “InputDevice”
    Identifier “ETouch”
    Driver “ETouch”
    Option “Device” “/dev/usb/hiddev0″
EndSection

Section “ServerLayout”
    Identifier “Layout01″
    InputDevice “ETouch” “SendCoreEvents”
EndSection

Pay attention only to the part mark with red color. Save it and restart your system.
If you type it correctly, there should be no problem. If you do encounter problems, boot your system using Recovery Mode ( Select it from Boot Menu by pressing ESC), and check the Xorg log at /var/log/Xorg.0.log
Read the log carefully and then make changes to Xorg.conf, restart your system again by either type:

 sudo reboot -n

or simply by pressing

Ctrl+Alt+Del

When you got things right, your GUI will load, you touch screen will function, BUT… still not calibrated

• STEP 4: Calibration

Before you proceed, please make sure you unplug any external monitor and set the permission correctly.
Go to /etc/etandt folder and run this

sudo /etc/etandt/Calibration

If you dont use sudo, you will get Segmentation Fault
You’ll see white screen with small crossed circle. Press the center of it as precise as possible. You can cancel anytime by pressing Ctrl+C

You’re done.

References:
http://ubuntuforums.org/showthread.php?t=677833
http://magictouch.com/support_usb_xd.html
http://www.qbik.ch/usb/devices/showdescr.php?id=4460
http://forum.ubuntu-fr.org/viewtopic.php?pid=3073235 (in French)
http://widi-atmono.blogspot.com/2009/11/install-driver-touch-screen-m912m-di.html

Ubuntu – How to Restart Freezed/Hang Gnome Panel

After upgrading to Jaunty, in Compiz sometimes I get a lot of Gnome-Panel Freezes, hovering the mouse cursor over it won’t trigger any response.

This freezes usually triggered by pressing Alt-F2.

So, the gnome-panel has to be restarted with this command:

killall gnome-panel

nohup gnome-panel –replace &

References:

http://mylinuxnotebook.blogspot.com/2008/09/restart-gnome-panel.html