OK, after manually updating my JDK three times, which is painstaking on my Jaunty Jackalope machine, I finally got the hang of how to do it. Or at least so I thought. So, basically, the steps are as follow:

1. Download the JDK bin installer from Sun's (or now, Oracle's) site.
2. Put it in /usr/lib/jvm and execute it, which basically will unzip it.
3. Rename the resulting directory to follow the naming patterns of other directories of JDK of other versions.
4. Create the .jinfo file. Just copy paste the .jinfo from other version, change the directory, change the name, and change the priority number. Newer version should be assigned less priority number. (The more the priority number is, the less prioritized it is).
5. Call update-java-alternatives -l and see if your newly installed java is listed. If so, then call update-java-alternatives -s , and then see that you don't get "Can not update" message. If you do, and that was what happened to me, solution in the end of this post.
6. Locate all java plugins for Mozilla and delete them all, save the ones under /usr/lib/jvm. Not surprisingly, there is a linux command called "locate" to do that. But funnily, it is only yesterday that I found out about it. So basically you would want to call this command (without the quotes): "locate libnpjp2.so libjavaplugin_oji.so pluginreg.dat", and then note the directory names because you are going to need them later, and then delete all of them but not the ones under /usr/lib/jvm.
7. Go to mozilla plugin directories (the ones you took note in step 6) and re-create the symlink to java plugin which is located in /lib/i386/libnpjp2.so. This is the command that you need (again, without the quotes): "ln -s /lib/i386/libnpjp2.so . " Remember to do this for each plugin that you deleted in step 6. That is, for each libnpjp2.so and libjavaplugin_oji.so file. And then check that you firefox has updated its plugin by doing step 8.
8. Re-start firefox and then in the address bar type: "about:plugins". (Without the quotes). Scroll and see that you have the Java (TM) plugin of the correct version.

If you did all the above steps and still something went awry, well... tinker, google, may God help you. Oh yeah, sometimes running update-java-alternatives gives you a horror message of "can not update...". It turned out that simply making .jinfo file sometimes does not update the information or... something to that effect. I do not really understand how this update mechanism works. But fortunately, someone from Ubuntu forum made a nifty shell script to execute to before calling update-java-alternatives and the script really saved my day. The following is the script:

#!/bin/sh
update-alternatives --install /usr/bin/ControlPanel ControlPanel /usr/lib/jvm/java-6-sun-1.6.0.20/jre/bin/ControlPanel 60
update-alternatives --install /usr/bin/java java /usr/lib/jvm/java-6-sun-1.6.0.20/jre/bin/java 60
update-alternatives --install /usr/bin/java_vm  java_vm /usr/lib/jvm/java-6-sun-1.6.0.20/jre/bin/java_vm 60
update-alternatives --install /usr/bin/javaws  javaws /usr/lib/jvm/java-6-sun-1.6.0.20/jre/bin/javaws 60
update-alternatives --install /usr/bin/jcontrol  jcontrol /usr/lib/jvm/java-6-sun-1.6.0.20/jre/bin/jcontrol 60
update-alternatives --install /usr/bin/keytool  keytool /usr/lib/jvm/java-6-sun-1.6.0.20/jre/bin/keytool 60
update-alternatives --install /usr/bin/pack200  pack200 /usr/lib/jvm/java-6-sun-1.6.0.20/jre/bin/pack200 60
update-alternatives --install /usr/bin/policytool  policytool /usr/lib/jvm/java-6-sun-1.6.0.20/jre/bin/policytool 60
update-alternatives --install /usr/bin/rmid  rmid /usr/lib/jvm/java-6-sun-1.6.0.20/jre/bin/rmid 60
update-alternatives --install /usr/bin/rmiregistry  rmiregistry /usr/lib/jvm/java-6-sun-1.6.0.20/jre/bin/rmiregistry 60
update-alternatives --install /usr/bin/unpack200  unpack200 /usr/lib/jvm/java-6-sun-1.6.0.20/jre/bin/unpack200 60
update-alternatives --install /usr/bin/orbd  orbd /usr/lib/jvm/java-6-sun-1.6.0.20/jre/bin/orbd 60
update-alternatives --install /usr/bin/servertool  servertool /usr/lib/jvm/java-6-sun-1.6.0.20/jre/bin/servertool 60
update-alternatives --install /usr/bin/tnameserv  tnameserv /usr/lib/jvm/java-6-sun-1.6.0.20/jre/bin/tnameserv 60
update-alternatives --install /usr/bin/HtmlConverter  HtmlConverter /usr/lib/jvm/java-6-sun-1.6.0.20/bin/HtmlConverter 60
update-alternatives --install /usr/bin/appletviewer  appletviewer /usr/lib/jvm/java-6-sun-1.6.0.20/bin/appletviewer 60
update-alternatives --install /usr/bin/apt  apt /usr/lib/jvm/java-6-sun-1.6.0.20/bin/apt 60
update-alternatives --install /usr/bin/extcheck  extcheck /usr/lib/jvm/java-6-sun-1.6.0.20/bin/extcheck 60
update-alternatives --install /usr/bin/idlj  idlj /usr/lib/jvm/java-6-sun-1.6.0.20/bin/idlj 60
update-alternatives --install /usr/bin/jar  jar /usr/lib/jvm/java-6-sun-1.6.0.20/bin/jar 60
update-alternatives --install /usr/bin/jarsigner  jarsigner /usr/lib/jvm/java-6-sun-1.6.0.20/bin/jarsigner 60
update-alternatives --install /usr/bin/java-rmi.cgi  java-rmi.cgi /usr/lib/jvm/java-6-sun-1.6.0.20/bin/java-rmi.cgi 60
update-alternatives --install /usr/bin/javac  javac /usr/lib/jvm/java-6-sun-1.6.0.20/bin/javac 60
update-alternatives --install /usr/bin/javadoc  javadoc /usr/lib/jvm/java-6-sun-1.6.0.20/bin/javadoc 60
update-alternatives --install /usr/bin/javah  javah /usr/lib/jvm/java-6-sun-1.6.0.20/bin/javah 60
update-alternatives --install /usr/bin/javap  javap /usr/lib/jvm/java-6-sun-1.6.0.20/bin/javap 60
update-alternatives --install /usr/bin/jconsole  jconsole /usr/lib/jvm/java-6-sun-1.6.0.20/bin/jconsole 60
update-alternatives --install /usr/bin/jdb  jdb /usr/lib/jvm/java-6-sun-1.6.0.20/bin/jdb 60
update-alternatives --install /usr/bin/jhat  jhat /usr/lib/jvm/java-6-sun-1.6.0.20/bin/jhat 60
update-alternatives --install /usr/bin/jinfo  jinfo /usr/lib/jvm/java-6-sun-1.6.0.20/bin/jinfo 60
update-alternatives --install /usr/bin/jmap  jmap /usr/lib/jvm/java-6-sun-1.6.0.20/bin/jmap 60
update-alternatives --install /usr/bin/jps  jps /usr/lib/jvm/java-6-sun-1.6.0.20/bin/jps 60
update-alternatives --install /usr/bin/jrunscript  jrunscript /usr/lib/jvm/java-6-sun-1.6.0.20/bin/jrunscript 60
update-alternatives --install /usr/bin/jsadebugd  jsadebugd /usr/lib/jvm/java-6-sun-1.6.0.20/bin/jsadebugd 60
update-alternatives --install /usr/bin/jstack  jstack /usr/lib/jvm/java-6-sun-1.6.0.20/bin/jstack 60
update-alternatives --install /usr/bin/jstat  jstat /usr/lib/jvm/java-6-sun-1.6.0.20/bin/jstat 60
update-alternatives --install /usr/bin/jstatd  jstatd /usr/lib/jvm/java-6-sun-1.6.0.20/bin/jstatd 60
update-alternatives --install /usr/bin/native2ascii  native2ascii /usr/lib/jvm/java-6-sun-1.6.0.20/bin/native2ascii 60
update-alternatives --install /usr/bin/rmic  rmic /usr/lib/jvm/java-6-sun-1.6.0.20/bin/rmic 60
update-alternatives --install /usr/bin/schemagen  schemagen /usr/lib/jvm/java-6-sun-1.6.0.20/bin/schemagen 60
update-alternatives --install /usr/bin/serialver  serialver /usr/lib/jvm/java-6-sun-1.6.0.20/bin/serialver 60
update-alternatives --install /usr/bin/wsgen  wsgen /usr/lib/jvm/java-6-sun-1.6.0.20/bin/wsgen 60
update-alternatives --install /usr/bin/wsimport  wsimport /usr/lib/jvm/java-6-sun-1.6.0.20/bin/wsimport 60
update-alternatives --install /usr/bin/xjc  xjc /usr/lib/jvm/java-6-sun-1.6.0.20/bin/xjc 60
update-alternatives --install /usr/bin/jexec  jexec /usr/lib/jvm/java-6-sun-1.6.0.20/jre/lib/jexec 60
update-alternatives --install /usr/lib/mozilla-firefox/plugins/libnpjp2.so libnpjp2.so /usr/lib/jvm/java-6-sun-1.6.0.20/jre/lib/i386/libnpjp2.so 60

Update Feb 20, 2011:
So just now I tried to update it again to JDK 6 update 24. I followed the above procedure, but my Firefox plugins still won't update. I did the following:

1. I deleted the Java plugin symlinks in mozilla firefox home directory and replace them with the new ones pointing to Java 6 u 24.
2. I deleted the Java plugin symlinks in mozilla firefox home directory and DIDN'T REPLACE THEM.

Well, that's all for now folks :) References:
http://java.sun.com/javase/6/webnotes/install/jre/manual-plugin-install-linux.html
http://ubuntuforums.org/showthread.php?t=571257
http://mrinvader.blogspot.com/2010/05/stupid-effing-java.html