highaltitudedev: 2011

Friday, July 8, 2011

URL Classloader

Recently I needed to supply a CXF WS client with a URL of the WSDL, but I needed to load it locally, from my classpath. I found a nifty solution on Stackoverflow to load classpath resources with a URL:

public static class Handler extends URLStreamHandler {
     /** The classloader to find resources from. */
     private final ClassLoader classLoader;

     public Handler(ClassLoader classLoader) {
         this.classLoader = classLoader;
     }

     @Override
     protected URLConnection openConnection(URL u) throws IOException {
         final URL resourceUrl = classLoader.getResource(u.getPath());
         return resourceUrl.openConnection();
     }
 }

Then create the URL object and hand off to the CXF Proxy

url = new URL(null, "classpath:my.wsdl", new Handler(this.getClass().getClassLoader()));
MyService service = new MyService(url);

Credit goes to Stephen for this awesome trick:
http://stackoverflow.com/questions/861500/url-to-load-resources-from-the-classpath-in-java

Saturday, July 2, 2011

Sheevaplug Wifi USB Adapter

I recently added a Wifi adapter to my Sheevaplug since I don't have ethernet in my office. I purchased the Rosewill wifi adapter from Newegg, after reviews seemed to indicate it plays well with Linux. After some searching I learned that there are two main Linux tools for wireless configuration: wireless-tools and wpasupplicant. I installed wpasupplicant since my wifi access point uses WPA-PSK security. You can use wireless-tools if you only need WEP (lesser security).

apt-get install wpasupplicant

Then generate the config

wpa_passphrase <essid> <passphrase>

This will generate something like

network={
ssid="rawr"
#psk="foofoofoo"
psk=4a06c8003af16578addc1400cb3fed969dfa94a07164236b886301d20d05fb74
}

I added

proto=WPA
key_mgmt=WPA-PSK

Save the config in /etc/wpa_supplicant.conf

Scan for networks

iwlist scan

Hopefully your wireless network appears.

Test wpasupplicant

wpa_supplicant -iwlan0 -c/etc/wpa_supplicant.conf -Dwext

and run ifconfig to see if it gets an ip address

If it's working, add to /etc/network/interfaces

auto wlan0
iface wlan0 inet dhcp
wpa-driver wext
wpa-conf /etc/wpa_supplicant.conf

And restart

/etc/init.d/networking restart

And you should have a wireless Sheevaplug.

Sheevaplug Power Supply Failure and Repair

My Sheevaplug's power supply failed recently, after less than a year of operation. Unfortunately this is a common occurrence with GlobalScales Sheevaplug, and with only a 30 warranty it's leaving many owners a bit peeved. Instead of purchasing a replacement power supply, which will likely fail again, I opted for an external (wall wart) power supply. I'm currently using a 5V/2A supply, which works, however the original power supply is 5V/3A. I ordered this 5V/3A power supply off Ebay. Beware this will take 3 or so weeks to arrive from China, but you can also order from US suppliers and pay just a bit more. You'll notice this adapter has a 2.1/5.5mm barrel connector but the Sheevaplug uses a Molex, so you'll need to cut the Molex wires and solder on a barrel adapter. I had one of these on hand, so I used it instead -- not exactly ideal as it is designed to mount on a pcb. To compensate for the poor choice in parts, I slathered it with hot glue to insulate the contacts. Be sure to identify the positive and negative connecters on the power supply, with a multimeter (mixing this up will probably destroy your plug).

I unscrewed the original power port and fed the barrel connector through and into the metal box where I connected it to the Molex wire.

I powered it on and the plug came back to life. Unfortunately my SD card seemed to have gotten corrupted, which I was booting off of, so I had to fallback to boot off the flash. The plug is working great now and hopefully it's even more reliable.

Tuesday, March 15, 2011

Spring Config with Multiple PropertyPlaceholderConfigurer(s)

The project I'm involved with is using the PropertyPlaceholderConfigurer feature of Spring to manage properties. This is convenient as it allows you to load properties from the file system, but fallback to default properties if the file does not exist. For example, use defaults in your development environment, but override the properties in QA and production.

<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer" id="propertyConfigurer">
        <property name="properties">
            <props>
                <prop key="hibernate.connection.url">jdbc:mysql://localhost/database</prop>
                <prop key="hibernate.connection.username">username</prop>
                <prop key="hibernate.connection.password">password</prop>
            </props>
        </property>
        
        <property name="locations">
            <list>
                <value>file:/app/hibernate.properties</value>
            </list>
        </property>
        
        <property name="ignoreResourceNotFound" value="true">
    </property>
</bean>

This was working excellently until I introduced another PropertyPlaceholderConfigurer in a different JAR. I started getting Spring exceptions on startup and the context would not load: BeanDefinitionStoreException: Invalid bean definition with name name.

After a fair amount of sleuthing, I found the solution on the Spring forums. The PropertyPlaceholderConfigurer that loads first must have the "order" property, with a "low" number". Additional PropertyPlaceholderConfigurer should have a higher "order":

<property name="order" value="100"/>

And the ignoreUnresolvablePlaceholders property must be set to true:

<property name="ignoreUnresolvablePlaceholders" value="true"/>

The other PropertyPlaceholderConfigurer needs to be updated as well. It needs a higher "order":

<property name="order" value="200"/>

And set ignoreUnresolvablePlaceholders to false, or omit it as false is the default:

<property name="ignoreUnresolvablePlaceholders" value="false"/>

Now I can import my properties and no collisions:

<import resource="classpath*:myProps.xml"></import>

Here's the Spring Forum thread and the credit goes to user pukomuko. Interesting that the one Spring expert on the thread didn't have the solution.

Wednesday, March 9, 2011

Null Characters and REST Calls

I have a CXF JAX-RS service that performs LDAP operations. While testing I discovered the following error:

org.xml.sax.SAXParseException: An invalid XML character (Unicode: 0x0) was found in the element content of the document.
    at javax.xml.bind.helpers.AbstractUnmarshallerImpl.createUnmarshalException(AbstractUnmarshallerImpl.java:315)
    at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.createUnmarshalException(UnmarshallerImpl.java:514)
    at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(UnmarshallerImpl.java:215)
    at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal(UnmarshallerImpl.java:184)
    at javax.xml.bind.helpers.AbstractUnmarshallerImpl.unmarshal(AbstractUnmarshallerImpl.java:137)
    at javax.xml.bind.helpers.AbstractUnmarshallerImpl.unmarshal(AbstractUnmarshallerImpl.java:194)

The LDAP code was throwing an AuthenticationException with a null character (0x0) in the message, when connecting to an Active Directory server:

javax.naming.AuthenticationException: [LDAP: error code 49 - 8009030C: LdapErr:DSID-0C09043E, comment: AcceptSecurityContext error, data 0, vece]

The null character was appearing after "vece" and before the bracket.

The exception message is returned by the REST service as a String field in a JAXB object. Then, when the client attempts to unmarshal the object, it fails since the null character is illegal XML. Actually characters 0-31 are illegal, and possibly others as well. Interesting that it had no problem marshaling the object to XML. So, while JAX-RS annotations do a great job in hiding the REST details, and making it look a bit like RPC, it's definitely not RPC and errors like these remind you of that.

Saturday, March 5, 2011

Sheevaplug Recovery

I had my Sheevaplug booting off a SD card, which was really nice since this gave me an 8GB filesystem, a significant increase from the default 512mb flash filesystem. At some point my SD card became corrupted and to make matters worse, my original flash filesystem was gone. Boot attempts would display the following:

Starting kernel ...

Uncompressing Linux... done, booting the kernel.

and then it would just hang.

Fortunately the Sheevaplug people provide a recovery tool. There are two parts to the recovery tool: usb drive and host. Some of the recovery tools needs to get loaded onto a usb drive and placed into the Sheevaplug's usb port (the big one, not the mini usb port). Then, the host part runs on either Windows or Linux and communicates over the usb cable. I already have Ubuntu on VMware Fusion (Mac OSX), so I'll use that since it is a Linux system we're dealing with, and the setup looks slightly easier.

I followed the instructions in the README. Step 3 says to "edit the ~/work/sheevaplug-installer/installer/uboot-custom.txt file to burn the correct MAC address according to the MAC address on the back of the Plug (change the ethaddr value)". I created this file and entered the address with colons: 00:50:43:01:D0:75 (it didn't say anything about the format).

In step 5, the files were already there from step 4, odd. Step 5d asks me to Init ramdisk (uImage.initrd). I only have file a initrd file, so I copied initrd to uImage.initrd.

After copying the files to the USB drive, it looks like

README.txt    uboot.bin
initrd     uImage     ubuntu-sheevaplug.sh
modules.tar.gz    uImage.initrd
rootfs.tar.gz    uboot-custom.txt

PHP is required (odd choice but whatever), so need to install that

sudo apt-get install php5
sudo apt-get install php5-cli

At first I got a libftdi error

****   Burning uboot and environment variables ... This will take few minutes ...
openocd/openocd: error while loading shared libraries: libftdi.so.1: cannot open shared object file: No such file or directory

so I installed all sorts of ftdi. I don't think all of these are necessary but they can't hurt.

sudo apt-get install libftdi1
sudo apt-get install libftdi-dev
sudo apt-get install libftdipp1

And run the script again:

sudo php5 runme.php nand

**** exec(modprobe ftdi_sio vendor=0x9e88 product=0x9e8f)WARNING: All config files need .conf: /etc/modprobe.d/vmware-tools, it will be ignored in a future release.

 **** Preparing environment variables file ...
reading uboot/uboot-env/uboot-dflt.txt
uboot/uboot-env/fw_setenv baudrate 115200
CRC read error on uboot-env.bin: Success
uboot/uboot-env/fw_setenv loads_echo 0
uboot/uboot-env/fw_setenv ipaddr 10.4.50.165
uboot/uboot-env/fw_setenv serverip 10.4.50.5
uboot/uboot-env/fw_setenv rootpath /mnt/ARM_FS/
uboot/uboot-env/fw_setenv netmask 255.255.255.0
uboot/uboot-env/fw_setenv stdin serial
uboot/uboot-env/fw_setenv stdout serial
uboot/uboot-env/fw_setenv stderr serial
uboot/uboot-env/fw_setenv console console=ttyS0,115200 mtdparts=nand_mtd:0xc0000@0\(uboot\)ro,0x1ff00000@0x100000\(root\)
uboot/uboot-env/fw_setenv mainlineLinux no
uboot/uboot-env/fw_setenv CASset min
uboot/uboot-env/fw_setenv enaMonExt no
uboot/uboot-env/fw_setenv enaCpuStream no
uboot/uboot-env/fw_setenv enaWrAllo no
uboot/uboot-env/fw_setenv pexMode RC
uboot/uboot-env/fw_setenv disL2Cache no
uboot/uboot-env/fw_setenv setL2CacheWT yes
uboot/uboot-env/fw_setenv disL2Prefetch yes
uboot/uboot-env/fw_setenv enaICPref yes
uboot/uboot-env/fw_setenv enaDCPref yes
uboot/uboot-env/fw_setenv sata_dma_mode yes
uboot/uboot-env/fw_setenv MALLOC_len 1
uboot/uboot-env/fw_setenv ethprime egiga0
uboot/uboot-env/fw_setenv netbsd_en no
uboot/uboot-env/fw_setenv vxworks_en no
uboot/uboot-env/fw_setenv bootargs_root root=/dev/nfs rw
uboot/uboot-env/fw_setenv bootargs_end :::DB88FXX81:eth0:none
uboot/uboot-env/fw_setenv image_name uImage
uboot/uboot-env/fw_setenv bootcmd tftpboot 0x2000000 \$\(image_name\)\;setenv bootargs \$\(console\) \$\(bootargs_root\) nfsroot=\$\(serverip\):\$\(rootpath\) ip=\$\(ipaddr\):\$\(serverip\)\$\(bootargs_end\) \$\(mvNetConfig\) \$\(mvPhoneConfig\)\;  bootm 0x2000000\;
uboot/uboot-env/fw_setenv standalone fsload 0x2000000 \$\(image_name\)\;setenv bootargs \$\(console\) root=/dev/mtdblock0 rw ip=\$\(ipaddr\):\$\(serverip\)\$\(bootargs_end\) \$\(mvPhoneConfig\)\; bootm 0x2000000\;
uboot/uboot-env/fw_setenv bootdelay 3
uboot/uboot-env/fw_setenv disaMvPnp no
uboot/uboot-env/fw_setenv ethaddr 00:50:43:4e:2a:14
uboot/uboot-env/fw_setenv ethmtu 1500
uboot/uboot-env/fw_setenv mvPhoneConfig mv_phone_config=dev0:fxs,dev1:fxs
uboot/uboot-env/fw_setenv mvNetConfig mv_net_config=\(00:11:88:0f:62:81,0:1:2:3\),mtu=1500
uboot/uboot-env/fw_setenv usb0Mode host
uboot/uboot-env/fw_setenv yuk_ethaddr 00:00:00:EE:51:81
uboot/uboot-env/fw_setenv nandEcc 1bit
uboot/uboot-env/fw_setenv netretry no
uboot/uboot-env/fw_setenv rcvrip 169.254.100.100
uboot/uboot-env/fw_setenv loadaddr 0x02000000
uboot/uboot-env/fw_setenv autoload no
uboot/uboot-env/fw_setenv enaAutoRecovery yes
uboot/uboot-env/fw_setenv ethact egiga0
reading uboot/uboot-env/uboot-nand-custom.txt
uboot/uboot-env/fw_setenv bootargs_root ubi.mtd=1 root=ubi0:rootfs rootfstype=ubifs
uboot/uboot-env/fw_setenv mtdpartitions mtdparts=orion_nand:0x400000@0x100000\(uImage\),0x1fb00000@0x500000\(rootfs\)
uboot/uboot-env/fw_setenv ethaddr 00:50:43:01:c1:e6
uboot/uboot-env/fw_setenv real_bootcmd setenv bootargs \$\(bootargs_console\) \$\(mtdpartitions\) \$\(bootargs_root\)\; nand read.e 0x00800000 0x00100000 0x00400000\; bootm 0x00800000
uboot/uboot-env/fw_setenv bootargs_console console=ttyS0,115200
uboot/uboot-env/fw_setenv bootcmd run recover1
uboot/uboot-env/fw_setenv recover1 setenv mainlineLinux yes\; setenv arcNumber 2097\; setenv bootcmd run recover2\; saveenv\; reset
uboot/uboot-env/fw_setenv recover2 run recover3\; setenv bootcmd \$\(real_bootcmd\)\; saveenv\; setenv bootargs \$\(bootargs_console\) \$\(mtdpartitions\) root=/dev/ram0 rw ramdisk=0x01100000,8M install_type=nand\; bootm 0x00800000 0x01100000
uboot/uboot-env/fw_setenv recover3 run recover4\; nand erase clean 0x00100000 0x00400000\; nand write.e 0x00800000 0x00100000 0x00400000
uboot/uboot-env/fw_setenv recover4 usb start\; fatload usb 0 0x00800000 uImage\; fatload usb 0 0x01100000 initrd

 **** Burning uboot and environment variables ... This will take few minutes ...
Open On-Chip Debugger 0.2.0 (2009-07-26-14:56) Release
$URL: http://svn.berlios.de/svnroot/repos/openocd/tags/openocd-0.2.0/src/openocd.c $
For bug reports, read http://svn.berlios.de/svnroot/repos/openocd/trunk/BUGS
2000 kHz
jtag_nsrst_delay: 200
jtag_ntrst_delay: 200
dcc downloads are enabled
Error: unable to open ftdi device: device not found
Runtime error, file "command.c", line 469:
     **** openocd FAILED
 **** Is the mini USB cable connected?

Actually yes, it is connected. I can connect to the plug with cu fine so I don't think it has anything to do with vmware.

Then, I find this post http://plugcomputer.org/plugforum/index.php?action=printpage;topic=920.0

erpel suggests "you might need to change the file: sheevaplug-installer-v1.0/uboot/openocd/config/interface/sheevaplug.cfg" ... "This is due to a changed vendor ID after 07/2009."

Not coincidently, my plug shipped right around that time.

As the post sugguests I ran lsusb and discover that my vid/pid are incorrect. I updated the config as follows and try again.

interface ft2232
ft2232_layout sheevaplug
#ft2232_vid_pid 0x9e88 0x9e8f
ft2232 vid_pid 0x0403 0x6010
#ft2232_device_desc "SheevaPlug JTAGKey FT2232D B"
jtag_khz 2000

This time it works!

sudo php5 runme.php nand

Update 7/2: This time on Ubuntu 10.x (previous recovery was on 9.x), I got a php root error:

PHP Notice:  Undefined index: OS in /home/andrew/sheevaplug-installer-v1.0/runme.php on line 7
PHP Notice:  Undefined index: USER in /home/andrew/sheevaplug-installer-v1.0/runme.php on

I'm running as root so this is a bug. I fixed by commenting out the root check:

#if ($_ENV["USER"] != 'root')
#die("You must run this as root\n");

as described here: http://plugcomputer.org/plugforum/index.php?action=printpage;topic=2068.0 (thanks to iffypug)

When I ran runme.php again I got a connection error. This was resolved by installing openocd and using that instead of the version that comes with the installer. I got this tip from iffypug's forum post.

apt-get install openocd
$ cd uboot/openocd
$ mv openocd openocd.bak
$ ln -s /usr/bin/openocd openocd

This time it completes but I get another php error:

U-boot should be up and running now. Open your console ...

PHP Notice:  Undefined variable: string_beeps in /home/andrew/sheevaplug-installer-v1.0/runme.php on line 86

We'll pretend this didn't happen. When I connect to the plug via serial, it's still doing stuff and I get a bunch of gobbledygook characters. See gobblygook section at the end of the post for the dramatic conclusion.

End update

**** exec(modprobe ftdi_sio vendor=0x9e88 product=0x9e8f)WARNING: All config files need .conf: /etc/modprobe.d/vmware-tools, it will be ignored in a future release.

**** Preparing environment variables file ...
reading uboot/uboot-env/uboot-dflt.txt
uboot/uboot-env/fw_setenv baudrate 115200
CRC read error on uboot-env.bin: Success
uboot/uboot-env/fw_setenv loads_echo 0
uboot/uboot-env/fw_setenv ipaddr 10.4.50.165
uboot/uboot-env/fw_setenv serverip 10.4.50.5
uboot/uboot-env/fw_setenv rootpath /mnt/ARM_FS/
uboot/uboot-env/fw_setenv netmask 255.255.255.0
uboot/uboot-env/fw_setenv stdin serial
uboot/uboot-env/fw_setenv stdout serial
uboot/uboot-env/fw_setenv stderr serial
uboot/uboot-env/fw_setenv console console=ttyS0,115200 mtdparts=nand_mtd:0xc0000@0\(uboot\)ro,0x1ff00000@0x100000\(root\)
uboot/uboot-env/fw_setenv mainlineLinux no
uboot/uboot-env/fw_setenv CASset min
uboot/uboot-env/fw_setenv enaMonExt no
uboot/uboot-env/fw_setenv enaCpuStream no
uboot/uboot-env/fw_setenv enaWrAllo no
uboot/uboot-env/fw_setenv pexMode RC
uboot/uboot-env/fw_setenv disL2Cache no
uboot/uboot-env/fw_setenv setL2CacheWT yes
uboot/uboot-env/fw_setenv disL2Prefetch yes
uboot/uboot-env/fw_setenv enaICPref yes
uboot/uboot-env/fw_setenv enaDCPref yes
uboot/uboot-env/fw_setenv sata_dma_mode yes
uboot/uboot-env/fw_setenv MALLOC_len 1
uboot/uboot-env/fw_setenv ethprime egiga0
uboot/uboot-env/fw_setenv netbsd_en no
uboot/uboot-env/fw_setenv vxworks_en no
uboot/uboot-env/fw_setenv bootargs_root root=/dev/nfs rw
uboot/uboot-env/fw_setenv bootargs_end :::DB88FXX81:eth0:none
uboot/uboot-env/fw_setenv image_name uImage
uboot/uboot-env/fw_setenv bootcmd tftpboot 0x2000000 \$\(image_name\)\;setenv bootargs \$\(console\) \$\(bootargs_root\) nfsroot=\$\(serverip\):\$\(rootpath\) ip=\$\(ipaddr\):\$\(serverip\)\$\(bootargs_end\) \$\(mvNetConfig\) \$\(mvPhoneConfig\)\;  bootm 0x2000000\;
uboot/uboot-env/fw_setenv standalone fsload 0x2000000 \$\(image_name\)\;setenv bootargs \$\(console\) root=/dev/mtdblock0 rw ip=\$\(ipaddr\):\$\(serverip\)\$\(bootargs_end\) \$\(mvPhoneConfig\)\; bootm 0x2000000\;
uboot/uboot-env/fw_setenv bootdelay 3
uboot/uboot-env/fw_setenv disaMvPnp no
uboot/uboot-env/fw_setenv ethaddr 00:50:43:4e:2a:14
uboot/uboot-env/fw_setenv ethmtu 1500
uboot/uboot-env/fw_setenv mvPhoneConfig mv_phone_config=dev0:fxs,dev1:fxs
uboot/uboot-env/fw_setenv mvNetConfig mv_net_config=\(00:11:88:0f:62:81,0:1:2:3\),mtu=1500
uboot/uboot-env/fw_setenv usb0Mode host
uboot/uboot-env/fw_setenv yuk_ethaddr 00:00:00:EE:51:81
uboot/uboot-env/fw_setenv nandEcc 1bit
uboot/uboot-env/fw_setenv netretry no
uboot/uboot-env/fw_setenv rcvrip 169.254.100.100
uboot/uboot-env/fw_setenv loadaddr 0x02000000
uboot/uboot-env/fw_setenv autoload no
uboot/uboot-env/fw_setenv enaAutoRecovery yes
uboot/uboot-env/fw_setenv ethact egiga0
reading uboot/uboot-env/uboot-nand-custom.txt
uboot/uboot-env/fw_setenv bootargs_root ubi.mtd=1 root=ubi0:rootfs rootfstype=ubifs
uboot/uboot-env/fw_setenv mtdpartitions mtdparts=orion_nand:0x400000@0x100000\(uImage\),0x1fb00000@0x500000\(rootfs\)
uboot/uboot-env/fw_setenv ethaddr 00:50:43:01:c1:e6
uboot/uboot-env/fw_setenv real_bootcmd setenv bootargs \$\(bootargs_console\) \$\(mtdpartitions\) \$\(bootargs_root\)\; nand read.e 0x00800000 0x00100000 0x00400000\; bootm 0x00800000
uboot/uboot-env/fw_setenv bootargs_console console=ttyS0,115200
uboot/uboot-env/fw_setenv bootcmd run recover1
uboot/uboot-env/fw_setenv recover1 setenv mainlineLinux yes\; setenv arcNumber 2097\; setenv bootcmd run recover2\; saveenv\; reset
uboot/uboot-env/fw_setenv recover2 run recover3\; setenv bootcmd \$\(real_bootcmd\)\; saveenv\; setenv bootargs \$\(bootargs_console\) \$\(mtdpartitions\) root=/dev/ram0 rw ramdisk=0x01100000,8M install_type=nand\; bootm 0x00800000 0x01100000
uboot/uboot-env/fw_setenv recover3 run recover4\; nand erase clean 0x00100000 0x00400000\; nand write.e 0x00800000 0x00100000 0x00400000
uboot/uboot-env/fw_setenv recover4 usb start\; fatload usb 0 0x00800000 uImage\; fatload usb 0 0x01100000 initrd

**** Burning uboot and environment variables ... This will take few minutes ...
Open On-Chip Debugger 0.2.0 (2009-07-26-14:56) Release
$URL: http://svn.berlios.de/svnroot/repos/openocd/tags/openocd-0.2.0/src/openocd.c $
For bug reports, read http://svn.berlios.de/svnroot/repos/openocd/trunk/BUGS
2000 kHz
jtag_nsrst_delay: 200
jtag_ntrst_delay: 200
dcc downloads are enabled
Error: unable to open ftdi device: device not found
Runtime error, file "command.c", line 469:
**** openocd FAILED
**** Is the mini USB cable connected?
me@ubuntu:~/Desktop/sheevaplug-installer-v1.0$ vi uboot/openocd/config/interface/sheevaplug.cfg 
me@ubuntu:~/Desktop/sheevaplug-installer-v1.0$ sudo php5 runme.php nand
**** exec(modprobe ftdi_sio vendor=0x9e88 product=0x9e8f)WARNING: All config files need .conf: /etc/modprobe.d/vmware-tools, it will be ignored in a future release.

**** Preparing environment variables file ...
reading uboot/uboot-env/uboot-dflt.txt
uboot/uboot-env/fw_setenv baudrate 115200
CRC read error on uboot-env.bin: Success
uboot/uboot-env/fw_setenv loads_echo 0
uboot/uboot-env/fw_setenv ipaddr 10.4.50.165
uboot/uboot-env/fw_setenv serverip 10.4.50.5
uboot/uboot-env/fw_setenv rootpath /mnt/ARM_FS/
uboot/uboot-env/fw_setenv netmask 255.255.255.0
uboot/uboot-env/fw_setenv stdin serial
uboot/uboot-env/fw_setenv stdout serial
uboot/uboot-env/fw_setenv stderr serial
uboot/uboot-env/fw_setenv console console=ttyS0,115200 mtdparts=nand_mtd:0xc0000@0\(uboot\)ro,0x1ff00000@0x100000\(root\)
uboot/uboot-env/fw_setenv mainlineLinux no
uboot/uboot-env/fw_setenv CASset min
uboot/uboot-env/fw_setenv enaMonExt no
uboot/uboot-env/fw_setenv enaCpuStream no
uboot/uboot-env/fw_setenv enaWrAllo no
uboot/uboot-env/fw_setenv pexMode RC
uboot/uboot-env/fw_setenv disL2Cache no
uboot/uboot-env/fw_setenv setL2CacheWT yes
uboot/uboot-env/fw_setenv disL2Prefetch yes
uboot/uboot-env/fw_setenv enaICPref yes
uboot/uboot-env/fw_setenv enaDCPref yes
uboot/uboot-env/fw_setenv sata_dma_mode yes
uboot/uboot-env/fw_setenv MALLOC_len 1
uboot/uboot-env/fw_setenv ethprime egiga0
uboot/uboot-env/fw_setenv netbsd_en no
uboot/uboot-env/fw_setenv vxworks_en no
uboot/uboot-env/fw_setenv bootargs_root root=/dev/nfs rw
uboot/uboot-env/fw_setenv bootargs_end :::DB88FXX81:eth0:none
uboot/uboot-env/fw_setenv image_name uImage
uboot/uboot-env/fw_setenv bootcmd tftpboot 0x2000000 \$\(image_name\)\;setenv bootargs \$\(console\) \$\(bootargs_root\) nfsroot=\$\(serverip\):\$\(rootpath\) ip=\$\(ipaddr\):\$\(serverip\)\$\(bootargs_end\) \$\(mvNetConfig\) \$\(mvPhoneConfig\)\;  bootm 0x2000000\;
uboot/uboot-env/fw_setenv standalone fsload 0x2000000 \$\(image_name\)\;setenv bootargs \$\(console\) root=/dev/mtdblock0 rw ip=\$\(ipaddr\):\$\(serverip\)\$\(bootargs_end\) \$\(mvPhoneConfig\)\; bootm 0x2000000\;
uboot/uboot-env/fw_setenv bootdelay 3
uboot/uboot-env/fw_setenv disaMvPnp no
uboot/uboot-env/fw_setenv ethaddr 00:50:43:4e:2a:14
uboot/uboot-env/fw_setenv ethmtu 1500
uboot/uboot-env/fw_setenv mvPhoneConfig mv_phone_config=dev0:fxs,dev1:fxs
uboot/uboot-env/fw_setenv mvNetConfig mv_net_config=\(00:11:88:0f:62:81,0:1:2:3\),mtu=1500
uboot/uboot-env/fw_setenv usb0Mode host
uboot/uboot-env/fw_setenv yuk_ethaddr 00:00:00:EE:51:81
uboot/uboot-env/fw_setenv nandEcc 1bit
uboot/uboot-env/fw_setenv netretry no
uboot/uboot-env/fw_setenv rcvrip 169.254.100.100
uboot/uboot-env/fw_setenv loadaddr 0x02000000
uboot/uboot-env/fw_setenv autoload no
uboot/uboot-env/fw_setenv enaAutoRecovery yes
uboot/uboot-env/fw_setenv ethact egiga0
reading uboot/uboot-env/uboot-nand-custom.txt
uboot/uboot-env/fw_setenv bootargs_root ubi.mtd=1 root=ubi0:rootfs rootfstype=ubifs
uboot/uboot-env/fw_setenv mtdpartitions mtdparts=orion_nand:0x400000@0x100000\(uImage\),0x1fb00000@0x500000\(rootfs\)
uboot/uboot-env/fw_setenv ethaddr 00:50:43:01:c1:e6
uboot/uboot-env/fw_setenv real_bootcmd setenv bootargs \$\(bootargs_console\) \$\(mtdpartitions\) \$\(bootargs_root\)\; nand read.e 0x00800000 0x00100000 0x00400000\; bootm 0x00800000
uboot/uboot-env/fw_setenv bootargs_console console=ttyS0,115200
uboot/uboot-env/fw_setenv bootcmd run recover1
uboot/uboot-env/fw_setenv recover1 setenv mainlineLinux yes\; setenv arcNumber 2097\; setenv bootcmd run recover2\; saveenv\; reset
uboot/uboot-env/fw_setenv recover2 run recover3\; setenv bootcmd \$\(real_bootcmd\)\; saveenv\; setenv bootargs \$\(bootargs_console\) \$\(mtdpartitions\) root=/dev/ram0 rw ramdisk=0x01100000,8M install_type=nand\; bootm 0x00800000 0x01100000
uboot/uboot-env/fw_setenv recover3 run recover4\; nand erase clean 0x00100000 0x00400000\; nand write.e 0x00800000 0x00100000 0x00400000
uboot/uboot-env/fw_setenv recover4 usb start\; fatload usb 0 0x00800000 uImage\; fatload usb 0 0x01100000 initrd

**** Burning uboot and environment variables ... This will take few minutes ...
Open On-Chip Debugger 0.2.0 (2009-07-26-14:56) Release
$URL: http://svn.berlios.de/svnroot/repos/openocd/tags/openocd-0.2.0/src/openocd.c $
For bug reports, read http://svn.berlios.de/svnroot/repos/openocd/trunk/BUGS
2000 kHz
jtag_nsrst_delay: 200
jtag_ntrst_delay: 200
dcc downloads are enabled
Info : JTAG tap: feroceon.cpu tap/device found: 0x20a023d3 (mfg: 0x1e9, part: 0x0a02, ver: 0x2)
Info : JTAG Tap/device matched
target state: halted
target halted in ARM state due to debug-request, current mode: Supervisor
cpsr: 0x000000d3 pc: 0xffff0000
MMU: disabled, D-Cache: disabled, I-Cache: disabled
0 0 1 0: 00052078
NAND flash device 'NAND 512MiB 3,3V 8-bit' found
successfully erased blocks 5 to 6 on NAND flash device 'NAND 512MiB 3,3V 8-bit'
wrote file uboot-env.bin to NAND flash 0 up to offset 0x000c0000 in 78.586975s
target state: halted
target halted in ARM state due to debug-request, current mode: Supervisor
cpsr: 0x000000d3 pc: 0xffff0000
MMU: disabled, D-Cache: disabled, I-Cache: disabled
0 0 1 0: 00052078
NAND flash device 'NAND 512MiB 3,3V 8-bit' found
successfully erased blocks 0 to 4 on NAND flash device 'NAND 512MiB 3,3V 8-bit'
wrote file uboot.bin to NAND flash 0 up to offset 0x00074000 in 298.123779s

**** U-boot should be up and running now. Open your console ...

And now I can log into my plug!

sudo chown uucp /dev/ttyUSB1 
me@ubuntu:~/Desktop/sheevaplug-installer-v1.0$ sudo cu -s 115200 -l /dev/ttyUSB1 
Connected.

Ubuntu 9.04 ubuntu ttyS0

ubuntu login:

I had to lookup the default root password: nosoup4u.

Next, I installed a prebuilt kernel so I can get usb-serial port support on the plug.

http://plugcomputer.org/plugwiki/index.php/Install_Prebuilt_Kernels_From_sheeva.with-linux.com

It says to update bootargs, only when I printenv bootargs I get:

## Error: "bootargs": not defined

It seems that UBoot was updated during the recovery process and bootargs is not longer a parameter. My Uboot version is now:

U-Boot 1.1.4 (Jul 19 2009 - 16:03:28) Marvell version: 3.4.19

It boots fine so I'm not going to worry too much at this point.

Now I'm going to modify the plug to boot off an 8GB SD Card. I will follow this guide http://www.plugcomputer.org/plugwiki/index.php/SD_Card_As_Root_File_System

I need to follow the "Changes for Alpha6 Beta" since I used the recovery tool. It says to set the following uboot parameter:

setenv bootargs_root root=b301

Then save and boot:

saveenv
boot

The boot console now shows:

EXT3 FS on mmcblk0p1, internal journal
EXT3-fs: mounted filesystem with writeback data mode.
VFS: Mounted root (ext3 filesystem) on device 179:1.

So I know it's booting off of the SD card. Furthermore, a 'df' shows:

root@pluggy:~# df
Filesystem           1K-blocks      Used Available Use% Mounted on
tmpfs                   256872         0    256872   0% /lib/init/rw
varrun                  256872        36    256836   1% /var/run
varlock                 256872         0    256872   0% /var/lock
udev                    256872       112    256760   1% /dev
tmpfs                   256872         0    256872   0% /dev/shm
rootfs                 7692876    528788   6773312   8% /

If I want to go back to booting off of NAND, it's just a matter of changing bootargs_root to

bootargs_root=ubi.mtd=1 root=ubi0:rootfs rootfstype=ubifs

Next up I plan to install the Sun (Oracle) JDK for ARM so I can run my Java apps.

Here's the output (as far as history would scroll) of my second recovery on Ubuntu 10.x (first was Ubuntu 9.x, which was much smoother

�ɵ��� 2�ɵ��ѥ���*Ʌ͕�ѕѕ� jT����ɵ��� 2�ɵ��ѥ���*Ʌ͕�jj��ѕ� jT����ɵ��� 2�ɵ��ѥ���*Ʌ͕������������ jj �� * �����ѕ� jT���: fobiformat: formatting e: formatting erasiformat: formatt % complet^C^C^CUBI: attached mtd1 to ubi0
UBI: MTD device name:            "rootfs"
UBI: MTD device size:            507 MiB
UBI: number of good PEBs:        4040
UBI: number of bad PEBs:         16
UBI: max. allowed volumes:       128
UBI: wear-leveling threshold:    4096
UBI: number of internal volumes: 1
UBI: number of user volumes:     0
UBI: available PEBs:             3996
UBI: total number of reserved PEBs: 44
UBI: number of PEBs reserved for bad PEB handling: 40
UBI: max/mean erase counter: 11/3
UBI: background thread "ubi_bgt0d" started, PID 549
UBI device number 0, total 4040 LEBs (521256960 bytes, 497.1 MiB), available 3996 LEBs (515579904 bytes, 491.7 MiB), LEB size 129024 bytes (126.0 KiB)
Set volume size to 515579904
Volume ID 0, size 3996 LEBs (515579904 bytes, 491.7 MiB), LEB size 129024 bytes (126.0 KiB), dynamic, name "rootfs", alignment 1
**** Mounting /dev/sda1
UBIFS: default file-system created
UBIFS: mounted UBI device 0, volume 0, name "rootfs"
UBIFS: file system size:   513644544 bytes (501606 KiB, 489 MiB, 3981 LEBs)
UBIFS: journal size:       25675776 bytes (25074 KiB, 24 MiB, 199 LEBs)
UBIFS: media format:       w4/r0 (latest is w4/r0)
UBIFS: default compressor: lzo
UBIFS: reserved for root:  4952683 bytes (4836 KiB)
**** Copying root filesystem. This will take few minutes
**** done.                                                                 [A^[[A^[[A
**** Copying kernel modules
**** done.                                                                 
**** Copying kernel
**** done.
**** Doing sync. This will take some time
**** done. Rebooting...
UBIFS: un-mount UBI device 0, volume 0
The system is going down NOW!
Sent SIGTERM to all processes
Sent SIGKILL to all processes
Requesting system reboot
Restarting system.
�
         __  __                      _ _
        |  \/  | __ _ _ ____   _____| | |
        | |\/| |/ _` | '__\ \ / / _ \ | |
        | |  | | (_| | |   \ V /  __/ | |
        |_|  |_|\__,_|_|    \_/ \___|_|_|
 _   _     ____              _
| | | |   | __ )  ___   ___ | |_ 
| | | |___|  _ \ / _ \ / _ \| __| 
| |_| |___| |_) | (_) | (_) | |_ 
 \___/    |____/ \___/ \___/ \__| 
 ** MARVELL BOARD: SHEEVA PLUG LE 

U-Boot 1.1.4 (Jul 19 2009 - 16:03:28) Marvell version: 3.4.19

U-Boot code: 00600000 -> 0067FFF0  BSS: -> 006CFB00

Soc: 88F6281 A0 (DDR2)
CPU running @ 1200Mhz L2 running @ 400Mhz
SysClock = 400Mhz , TClock = 200Mhz 

DRAM CAS Latency = 5 tRP = 5 tRAS = 18 tRCD=6
DRAM CS[0] base 0x00000000   size 256MB 
DRAM CS[1] base 0x10000000   size 256MB 
DRAM Total size 512MB  16bit width
Addresses 8M - 0M are saved for the U-Boot usage.
Mem malloc Initialization (8M - 7M): Done
NAND:512 MB
Flash:  0 kB

CPU : Marvell Feroceon (Rev 1)

Streaming disabled 
Write allocate disabled


USB 0: host mode
PEX 0: interface detected no Link.
Net:   egiga0 [PRIME], egiga1
Hit any key to stop autoboot:  0 

NAND read: device 0 offset 0x100000, size 0x400000

Reading data from 0x4ff800 -- 100% complete.
 4194304 bytes read: OK
## Booting image at 00800000 ...
   Image Name:   Linux-2.6.30.2
   Created:      2009-07-23   1:53:36 UTC
   Image Type:   ARM Linux Kernel Image (uncompressed)
   Data Size:    2620440 Bytes =  2.5 MB
   Load Address: 00008000
   Entry Point:  00008000
   Verifying Checksum ... OK
OK

Starting kernel ...

Uncompressing Linux.......................................................................................................................................................................... done, booting the kernel.
Linux version 2.6.30.2 (kelly@speedy) (gcc version 4.2.1) #11 PREEMPT Wed Jul 22 19:53:31 MDT 2009
CPU: Feroceon 88FR131 [56251311] revision 1 (ARMv5TE), cr=00053977
CPU: VIVT data cache, VIVT instruction cache
Machine: Marvell SheevaPlug Reference Board
Memory policy: ECC disabled, Data cache writeback
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 130048
Kernel command line: console=ttyS0,115200 mtdparts=orion_nand:0x400000@0x100000(uImage),0x1fb00000@0x500000(rootfs) ubi.mtd=1 root=ubi0:rootfs rootfstype=ubifs
NR_IRQS:114
PID hash table entries: 2048 (order: 11, 8192 bytes)
Console: colour dummy device 80x30
Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
Memory: 256MB 256MB = 512MB total
Memory: 513408KB available (4728K code, 998K data, 140K init, 0K highmem)
SLUB: Genslabs=11, HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
Calibrating delay loop... 1192.75 BogoMIPS (lpj=5963776)
Mount-cache hash table entries: 512
CPU: Testing write buffer coherency: ok
net_namespace: 980 bytes
NET: Registered protocol family 16
Kirkwood: MV88F6281-A0, TCLK=200000000.
Feroceon L2: Enabling L2
Feroceon L2: Cache support initialised.
Kirkwood: Gating clock using mask 0x1ac224
bio: create slab  at 0
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
cfg80211: Using static regulatory domain info
cfg80211: Regulatory domain: US
 (start_freq - end_freq @ bandwidth), (max_antenna_gain, max_eirp)
 (2402000 KHz - 2472000 KHz @ 40000 KHz), (600 mBi, 2700 mBm)
 (5170000 KHz - 5190000 KHz @ 40000 KHz), (600 mBi, 2300 mBm)
 (5190000 KHz - 5210000 KHz @ 40000 KHz), (600 mBi, 2300 mBm)
 (5210000 KHz - 5230000 KHz @ 40000 KHz), (600 mBi, 2300 mBm)
 (5230000 KHz - 5330000 KHz @ 40000 KHz), (600 mBi, 2300 mBm)
 (5735000 KHz - 5835000 KHz @ 40000 KHz), (600 mBi, 3000 mBm)
cfg80211: Calling CRDA for country: US
NET: Registered protocol family 2
IP route cache hash table entries: 16384 (order: 4, 65536 bytes)
TCP established hash table entries: 65536 (order: 7, 524288 bytes)
TCP bind hash table entries: 65536 (order: 6, 262144 bytes)
TCP: Hash tables configured (established 65536 bind 65536)
TCP reno registered
NET: Registered protocol family 1
JFFS2 version 2.2. (NAND) © 2001-2006 Red Hat, Inc.
JFS: nTxBlock = 4012, nTxLock = 32100
msgmni has been set to 1003
alg: No test for stdrng (krng)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
Serial: 8250/16550 driver, 2 ports, IRQ sharing disabled
serial8250.0: ttyS0 at MMIO 0xf1012000 (irq = 33) is a 16550A
console [ttyS0] enabled
brd: module loaded
loop: module loaded
Driver 'sd' needs updating - please use bus_type methods
MV-643xx 10/100/1000 ethernet driver version 1.4
mv643xx_eth smi: probed
net eth0: port 0 with MAC address 00:50:43:01:c1:e6
libertas_sdio: Libertas SDIO driver
libertas_sdio: Copyright Pierre Ossman
NAND device: Manufacturer ID: 0xad, Chip ID: 0xdc (Hynix NAND 512MiB 3,3V 8-bit)
Scanning device for bad blocks
Bad eraseblock 1667 at 0x00000d060000
Bad eraseblock 1675 at 0x00000d160000
Bad eraseblock 1683 at 0x00000d260000
Bad eraseblock 1691 at 0x00000d360000
Bad eraseblock 1699 at 0x00000d460000
Bad eraseblock 1707 at 0x00000d560000
Bad eraseblock 1715 at 0x00000d660000
Bad eraseblock 1723 at 0x00000d760000
Bad eraseblock 3715 at 0x00001d060000
Bad eraseblock 3723 at 0x00001d160000
Bad eraseblock 3731 at 0x00001d260000
Bad eraseblock 3739 at 0x00001d360000
Bad eraseblock 3747 at 0x00001d460000
Bad eraseblock 3755 at 0x00001d560000
Bad eraseblock 3763 at 0x00001d660000
Bad eraseblock 3771 at 0x00001d760000
2 cmdlinepart partitions found on MTD device orion_nand
Creating 2 MTD partitions on "orion_nand":
0x000000100000-0x000000500000 : "uImage"
0x000000500000-0x000020000000 : "rootfs"
UBI: attaching mtd1 to ubi0
UBI: physical eraseblock size:   131072 bytes (128 KiB)
UBI: logical eraseblock size:    129024 bytes
UBI: smallest flash I/O unit:    2048
UBI: sub-page size:              512
UBI: VID header offset:          512 (aligned 512)
UBI: data offset:                2048
UBI: attached mtd1 to ubi0
UBI: MTD device name:            "rootfs"
UBI: MTD device size:            507 MiB
UBI: number of good PEBs:        4040
UBI: number of bad PEBs:         16
UBI: max. allowed volumes:       128
UBI: wear-leveling threshold:    4096
UBI: number of internal volumes: 1
UBI: number of user volumes:     1
UBI: available PEBs:             0
UBI: total number of reserved PEBs: 4040
UBI: number of PEBs reserved for bad PEB handling: 40
UBI: max/mean erase counter: 14/3
UBI: background thread "ubi_bgt0d" started, PID 443
ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
orion-ehci orion-ehci.0: Marvell Orion EHCI
orion-ehci orion-ehci.0: new USB bus registered, assigned bus number 1
orion-ehci orion-ehci.0: irq 19, io mem 0xf1050000
orion-ehci orion-ehci.0: USB 2.0 started, EHCI 1.00
usb usb1: configuration #1 chosen from 1 choice
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 1 port detected
Initializing USB Mass Storage driver...
usbcore: registered new interface driver usb-storage
USB Mass Storage support registered.
usbcore: registered new interface driver ums-datafab
usbcore: registered new interface driver ums-freecom
usbcore: registered new interface driver ums-jumpshot
usbcore: registered new interface driver ums-sddr09
usbcore: registered new interface driver ums-sddr55
mice: PS/2 mouse device common for all mice
rtc-mv rtc-mv: rtc core: registered rtc-mv as rtc0
i2c /dev entries driver
cpuidle: using governor ladder
cpuidle: using governor menu
sdhci: Secure Digital Host Controller Interface driver
sdhci: Copyright(c) Pierre Ossman
mmc0: mvsdio driver initialized, lacking card detect (fall back to polling)
Registered led device: plug:green:health
mv_xor_shared mv_xor_shared.0: Marvell shared XOR driver
mv_xor_shared mv_xor_shared.1: Marvell shared XOR driver
mv_xor mv_xor.0: Marvell XOR: ( xor cpy )
mv_xor mv_xor.1: Marvell XOR: ( xor fill cpy )
mv_xor mv_xor.2: Marvell XOR: ( xor cpy )
mv_xor mv_xor.3: Marvell XOR: ( xor fill cpy )
usbcore: registered new interface driver usbhid
usbhid: v2.6:USB HID core driver
oprofile: using timer interrupt.
TCP cubic registered
NET: Registered protocol family 17
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
lib80211: common routines for IEEE802.11 drivers
rtc-mv rtc-mv: setting system clock to 2011-07-03 16:49:35 UTC (1309711775)
UBIFS: mounted UBI device 0, volume 0, name "rootfs"
UBIFS: file system size:   513644544 bytes (501606 KiB, 489 MiB, 3981 LEBs)
UBIFS: journal size:       25675776 bytes (25074 KiB, 24 MiB, 199 LEBs)
UBIFS: media format:       w4/r0 (latest is w4/r0)
UBIFS: default compressor: lzo
UBIFS: reserved for root:  4952683 bytes (4836 KiB)
VFS: Mounted root (ubifs filesystem) on device 253:1.
Freeing init memory: 140K
usb 1-1: new high speed USB device using orion-ehci and address 2
usb 1-1: configuration #1 chosen from 1 choice
scsi0 : SCSI emulation for USB Mass Storage devices
 * Filesystem type 'fusectl' is not supported. Skipping mount.
 * Setting preliminary keymap...                                         [ OK ] 
 * Starting kernel event manager...                                      [ OK ] 
 * Loading hardware drivers...                                           [ OK ] 
 * Loading kernel modules...                                             [ OK ]  * Loading manual drivers...       
 * Setting kernel variables (/etc/sysctl.conf)...                        [ OK ] 
 * Setting kernel variables (/etc/sysctl.d/10-console-messages.conf)...  [ OK ] 
 * Setting kernel variables (/etc/sysctl.d/10-network-security.conf)...  [ OK ] 
 * Activating swap...                                                    [ OK ] 
 * Starting early crypto disks...                                        [ OK ] 
 * Starting remaining crypto disks...                                    [ OK ] 
 * Checking file systems...                                                     fsck 1.41.4 (27-Jan-2009)
                                                                         [ OK ]
 * Mounting local filesystems...                                         [ OK ] 
 * Activating swapfile swap...                                           [ OK ] 
 * Configuring network interfaces...                                     [ OK ] 
 * Setting up console font and keymap...                                 [ OK ] 
 * Starting system log daemon...                                                sd 0:0:0:0: [sda] Assuming drive cache: write through
sd 0:0:0:0: [sda] Assuming drive cache: write through
                                                                         [ OK ]
                                                                                * Starting kernel log daemon...       
Ubuntu 9.04 ubuntu ttyS0

ubuntu login:                                                            [ OK ]

Yay, I can has login

Tuesday, March 1, 2011

HttpClient Cookie Path Validation

I noticed some unusual behavior while using HttpClient 4 (version is important since it's a complete rewrite/namespace change from version 3) to log into a website. The login was failing and I saw the following message in the log (I've changed names to protect the innocent).

2011-02-25 16:41:27,154 [pool-1-thread-2] WARN  org.apache.http.client.protocol.ResponseProcessCookies[126] - Cookie rejected: "[version: 0][name: tracker][value: +][domain: .rawr.com][path: /splash][expiry: Thu Feb 25 16:41:27 MST 2010]". Illegal path attribute "/splash". Path of origin: "/login.php"

In this case the server is setting a cookie with an invalid path and HttpClient is properly rejecting the cookie. According to the RFC spec, a server must set the cookie path using the URL path, or an ancestor path. For example, when accessing http://foo.com/a/b/, the server may set a cookie with path /a/b/, /a, or /, but not /a/c. But since most all browsers (I only checked Firefox) ignore the spec and accept the cookie, I need HttpClient to do the same.

It's important to note that I'm using the browser compatibility cookie policy, but apparently it is missing some compatibility:

client.getParams().setParameter(ClientPNames.COOKIE_POLICY,  CookiePolicy.BROWSER_COMPATIBILITY);

Diving into the HttpClient code reveals that the cookie is being rejected by BasicPathHandler.java. This class is registered by BrowserCompatSpec.java (the implementation for CookiePolicy.BROWSER_COMPATIBILITY). In order to accept the cookie, it was necessary to subclass BrowserCompatSpec.java and and overwrite the existing path handler with an implementation that performs no validation:

HtmlUnitBrowserCompatCookieSpec() {
     super();
     
     final BasicPathHandler pathHandler = new BasicPathHandler() {
      @Override
      public void validate(final Cookie cookie, final CookieOrigin origin) throws MalformedCookieException {
       // nothing, browsers seem not to perform any validation
      }
     };
     registerAttribHandler(ClientCookie.PATH_ATTR, pathHandler);
    }

Credit goes to HtmlUnit for the solution.

Monday, February 21, 2011

Retries with the Spring HTTP Invoker

I've been doing work on a project that uses Spring Remoting with the HTTP Invoker. It makes remoting a piece of cake, and since it's HTTP, it's almost trivial to introduce a load balancer for HA/Scalability. I didn't realize that it was supporting retries until I saw this in the logs:

2011-02-07 19:59:37,541 [http-8080-2] INFO  org.apache.commons.httpclient.HttpMethodDirector[439] - I/O exception (java.net.ConnectException) caught when processing request: Connection refused
2011-02-07 19:59:37,541 [http-8080-2] INFO  org.apache.commons.httpclient.HttpMethodDirector[445] - Retrying request

I was a bit surprised to discover this, since it is not documented, and I was about to implement my own retry logic. The retry support actually has nothing to do with Spring Remoting, as it is built into HttpClient, which is used by the CommonsHttpInvokerRequestExecutor. By default, the HttpClient code will attempt up to three retries. The retry code is on line 433 of HttpMethodDirector.java (v 1.34 2005/01/14 19:40:39 olegk). The code is smart about what type of exceptions it will retry. The most typical exception that would get retried is an IOException (e.g. java.net.ConnectException). The retry code does not involve any delays, which if it did would be a big reason to disable it. But the code can block up to the connect timeout, under certain conditions, for example if the server is under heavy load. This of course translates to a potential 3*timeout delay. If you would like to suppress retries, or change the retry value, here's how to do it. Note, you will need to create the invoker programmatically to do this.

HttpInvokerProxyFactoryBean fb = new HttpInvokerProxyFactoryBean();

CommonsHttpInvokerRequestExecutor client = new CommonsHttpInvokerRequestExecutor();

client.getHttpClient().getParams().getDefaults().setParameter(HttpMethodParams.RETRY_HANDLER, new  DefaultHttpMethodRetryHandler(0, false));

fb.setHttpInvokerRequestExecutor(client);

The first argument to DefaultHttpMethodRetryHandler is the retry value. You could also configure the invoker in the Spring config, and then get a reference via the Spring context. Alternatively, you can use SimpleHttpInvokerRequestExecutor instead of CommonsHttpInvokerRequestExecutor. The former uses the java.net.HttpURLConnection, with no retry support, while the latter uses Apache HttpClient.

This is a good time to set your connection and read timeouts

client.getHttpConnectionManager().getParams().setConnectionTimeout(10000);
client.setReadTimeout(20000);

There, that's it. I hope. Note: I didn't exactly test this, since retries seem to be a good thing for me, however, it seems like it should work.