Using HTTP Over Socket and HTTP Proxy in J2ME Applications
As you know, MIDP version 2 (in fact, CLDC 1.x) offers very limited networking support. While trying to come up with a very simple, highly portable and as much flexible as possible networking API, Sun has missed a couple of important things. One of them is the ability to open HTTP connections through a proxy server.
In this article I tried as much as I can not to name the phone manufacturers and particular phone models and firmware releases.This is not the goal of this article to provide a (long ;) ) list of known bugs affecting them.
Quick links
If you ever had to do any networking in MIDP, you most likely used HttpConnection. This class is very easy to use, it does all the dirty job related to the HTTP protocol, provides easy access to the headers and the payload. And, most importantly, it usually works, at least if your phone is correctly configured. However, here is a small part of the long list of problems I had with using HttpConnection class in my past mobile experience:
- It gives no control on the way the connection is established. It is up to the operator or even the user who configured the phone. It is very easy to redirect the connection somewhere else for whoever really wants it.
- MIDP 2 does not allow you to use an HTTP proxy
- MIDP 2 does not support all the bells and whistles of HTTP 1.1. For example, there is no way you can receive multipart payloads. And some phones have problems supporting chuked encoding through HttpConnection.
- It usually works over WAP 1.x. There are quite few phones that can actually do it over WAP 2.x, i.e. over TCP. As result, you have all the fun associated with going through the WAP gateway (although, sometimes it is exactly what you want).
- I have seen only one phone that actually could use HTTP keep-alive with HttpConnection, from my favorite phone manufacturer - Nokia. But you absolutely had to configure the phone using special equipment in order to have this. HTTP keep-alive allows to build much faster network applications and reduces the network overhead caused bu the multiple connections being established and closed in short time if your application has to do several consecutive requests.
- At least half of the phones I worked with had problems with HTTP implementation. I have seen really strange cases - like duplicating the headers and their values, filtering out certain (quite useful!) HTTP headers, having some headers and values hardcoded, crashing when receiving certain headers, messing with the upper/lower case in the header names and values etc etc etc…
- There are phones (I will not mention the well-known brand) that are trying to use DNS to resolve a hostname even if you provide it as IP address :).
Although using of the HTTP over socket seems to be attractive, there is a number of limitations that have to be taken into account before deciding to go this way.
First of all, not every phone can do it. And not for any operator. You may be not allowed (by using various mechanisms like default security policy or firewalls) to use anything except WAP 1.x. The operator may charge you differently for using direct connections bypassing the WAP gateway. The phone may have, at the end, completely broken TCP stack.
Even if you technically can use the socket, you will have to implement at least a reasonable subset of HTTP protocol to be able to properly build the outgoing request and parse the response. Here is just a brief list of what you will have to take care about:
- Generating the proper request headers
- Parsing the response header taking into account all possible (meaningful for your application) combinations of the headers, extracting the values, validating every single piece of it. This part is usually simple if your application is talking only to your server so you have full control on what gets sent back.
- Managing the underlying network connections, handling the timeouts, implementing keep-alive logic
- Using buggy implementations of the SocketConnection
I have already mentioned the security policy. HTTP and socket connections require different permissions, thus they may have different security policy entries associated with them. On most of the phones it means that an unsigned application using the sockets will trigger the annoying security popups every time it opens new connection (using keep-alive may help a bit but at some point you will have to close the connection).
Some phones are not able to have more than one open TCP connection established. Even worse, your application may not be the only one running at certain point on this phone. So you may be unable to open the connection that you considered always available. Or even worse…
OK, I think I have scared you enough. If you are still ready to try, lets try to build an application that connects to a Web server through HTTP proxy over socket connection.
Lets try to build a simple application that will:
- have the HTTP proxy configurable through the JAD attributes
- prompt the user for the URL he/she wants to access
- will do GET request to the specified URL and fully receive the response header and the body
We will be using KToolbar utility from Sun Wireless Toolkit. Running of this application in the simulator will require a proxy server (Squid or even simple Apache Web Server configured to allow the proxy requests). Running it on a real device will require to know either operator's proxy server host name and port number or any public HTTP proxy server host name and port number (I doubt they still exist ;) ).
Lets skip the simple UI part and concentrate on the real stuff - the communication thread. It is very important in CLDC to do network I/O in a separate thread, you should never do any networking in the UI thread. On some smart phones (like B… ;) ) it is the known and reliable way to cause a nice deadlock.
First we will get the proxy parameters from the JAD attributes:
49 public void run() {
50 String proxyHost = getAppProperty("HTTP-Proxy-Host");
51 String proxyPort = getAppProperty("HTTP-Proxy-Port");
52 SocketConnection conn = null;
Note: this code does not perform all the possible parameter validation. J2ME programming patterns is a good topic for another (long) article.
57 // opening the socket connection 58 conn = (SocketConnection)Connector.open( 59 "socket://" + proxyHost + ":" + proxyPort);
In order to open a stream connection you need to use "socket://" protocol scheme followed by the host name or IP address and the port number (separated with ":"). For more information you should refer to the javadocs for the javax.microedition.io.Connector class.
60 conn.setSocketOption(SocketConnection.DELAY, 1); 61 conn.setSocketOption(SocketConnection.LINGER, 0); 62 conn.setSocketOption(SocketConnection.RCVBUF, 8192); 63 conn.setSocketOption(SocketConnection.SNDBUF, 512);
Setting certain socket option may help you to improve the performance of your network connections (for your particular application). However, some of these options do not work as expected (or do not work at all) on some phones.
147 private String getRequestHeader() {
148 StringBuffer buf = new StringBuffer(512);
149 String targetURL = mURL.getString();
150 buf.append("GET http://" + targetURL + " HTTP/1.1\n");
...
179 buf.append("User-Agent: MIDP2.0\n");
180 buf.append('\n');
181 return buf.toString();
It is a good idea to build the request header in a buffer and the write it into the output stream in one shot. At least you will do everything you can to ensure that the phone sends as few TCP packets as possible to the server at the beginning. As you can see, this method sets a number of parameters, some of them are optional. The request is built for making the HTTP request through a proxy server.
The next part is quite annoying - to read the input stream, separate the response header from the response body (if it is provided). I have seen at least a dozen of various implementations of this code and I do not claim that my implementation is efficient. It should be at least easy to read and understand. Also, if you have full control on the server you are connecting to, you can optimize the HTTP protocol parser so it will assume certain restrictions for the response header (for example, you will ensure that Content-length header is always provided).
79 while(bytesToGo > 0) {
80 int count = is.read(buf);
81 if (count == -1) {
82 // end-of-stream
83 break;
84 }
...
89 if (numLFs == 2 && numCRs == 2) {
90 // end of HTTP 1.1 header (ascii)
91 String hdrText = new String(headerStream.toByteArray());
92 responseStatus = Integer.parseInt(hdrText.substring(9, 12));
93 int clPos = hdrText.toLowerCase().indexOf("content-length: ");
94 if (clPos == -1) {
...
And finally, it is very important in J2ME programming to carefully close all the open connections and associated streams. Unlike on the desktops, the mobile Java implementations usually have limited amount of resources available and some platforms cannot properly release the resources that have not been closed even after the application exits. And, at the end, you should always release the resources in your code.
129 } finally {
130 try {
131 if (is != null) {
132 is.close();
133 }
134 if (os != null) {
...
Here is how it looks like when the application is running in the simulator:
You can download the full MIDlet code here.
### References - Java ME Technology API Documentation - Hypertext Transfer Protocol – HTTP/1.1 (specification)
blog comments powered by Disqus