Outline · [ Standard ] · Linear+ Webserver with Ajax and XML

[Darklord]

Ok, as promissed, im reporting on what i have done.

Here is the code for the prototype i have been working on, to test the concept and check some of the changes needed to implement it.

The code isnt meant to be added, just for checking and commenting; it was done against the code on SourceForge cvs (public, of course).

Following are some notes of implementation, covering what i have found out, done and thought about.

[Darklord]

Implementation Notes, part 1

After some preliminary research on the topic of ajax, decided to start with implementing a prototype that could show some of the benefits and problems with making the webserver ajax aware; this prototype would also permit some comments that could drive the rest of the development.

In this way, i decided to restrit the implementation to the stats bar, which already has several diferent changing parts (bars, value fields, etc) from several parts of emule.

In my research, i found out that there are multiple frameworks already to help develop ajax aplications (being the hot thing of the moment and all); however, its still a not very mature field, and since i has also doing this as a learning exercise, i decided not to use any of those, and implement everything on my own.
This means that it may not be very robust, and its reinventing the wheel, so thinking of choosing a framework to work with and reimplementing the code will probably be a necessary step.

As for data transfer format: a structured format for data is necessary, so simple html or text wasnt gonna make it; there are two main choices, XML and JSON (kinda like serialized javascript), but not knowing this at the time, i went with the one most associated with ajax, XML (the X on ajaX), and glad i did.

As the webserver as it was couldnt serve XML, i made the necessary changes (an ugly and imcomplete hack, the webserver needs serious refactoring); when done, it hit me with the new possibilities of this new interface:

- possibility of any XML aware program being able to monitor and control emulePlus easily; this includes IM programs, IRC programs, scripts, whatever...

- possibility of implementing true templates, using xml and xslt; bundling an archive of images and a xslt file, its possible to create pretty much any front end to the webserver from the browser, as heavy or lean as one wants, AND removing all the logic from the webserver, transfering it to the client browser; all the webserver gives is the data and processes the commands.

[Darklord]

Implementation Notes, part 2

For now, it still needs to login from the webserver to access the XML (to get the session id); a XML login is needed, but the changes were not worth it for a prototype; an also implemented http response with code 403 "Forbidden" is returned if a valid session id isnt given.

For the webserver to know that a XML response is wanted, a "channel" parameter is passed in the url with the value XML.

Also, to get the info, a "request" parameter is passed; there are two requests implemented, "testXML", for testing purposes, and "getStats", that gives the info needed for the bar (and some more).

So, the url to get the xml for the stats would be like this:

CODE

http://[webserver url]/?ses=[valid session id]&channel=xml&request=getStats

(case sensitive)

This is the request used to update the stats bar.

The format of the XML returned is also something that needs to be discussed in a future step.

A problem found was that the locale on my machine would interfere with the javascript notation expected (the simbol for decimals and thousands were switched); the easiest solution found was changing the application locale to english_USA while composing the XML reply, and restoring it after.

Another one, related, was with the translated strings used for the webserver; to process the data, a single language needs to be used, but this brings the problem with updating some of the fields; more on this below; the solution adopted was using english as the chosen language of the data.

This post has been edited by Darklord: Jun 5 2006, 22:20

[Fuxie - DK]

Here is the code for the prototype i have been working on

When I try to open the zip file (windows integrated ZIP support) I'm being told the file is corrupt..

[Darklord]

Implementation Notes, part 3

After the webserver stuff, i turned to the template; the XP template was used as a starting point.

First, i added all the javascript code in the template itself, but given the size of the template, i soon became tired of trying to find my code among all that; noticing the refernces to some external js scripts, found that the webserver can serve those, so i moved almost all the javascript code to another file (MUCH better);.
There was the need to force the browser to update the cache version, and i suppose thats the reason for the monstrous template, but was better than the alternative, for development.

I didnt have much previous experience with proggraming with javascript, and programming with ajax is somewhat more advanced that the usual js programming; so i hit the "debug using alert() hell" pretty fast.
Not wanting to go the debugger way (a recent book i read on C# says, quite suprisingly to me, that its the last resource one should take), i search for logging solutions, and settled on log4js, which gives a very usefull logging console, and is compatible with multiple browsers.
I removed the logging statements from the code on the archive.

What does the eMulePlus team use for debugging javascript?

As already said, no framework was used, its all custom made code.

[Darklord]

Implementation Notes, part 4

I "hijacked" the reload system and changed it for my purposes; one thing changed were the intervals, from 2m to 1 (changed in the template).
However, the system used isnt very good, since its just a series of timouts, a better event manager is necessary when theres need to more requests.

I created some functions to access more or less transparently the data on the XML response; however the approach used, of acessing the data one at a time, is extremely poor, since the parsing and retrieval of data from the XML is very consuming, in terms of time and cpu (but its not apparent in the prototype, so i let it be); this is one of the reasons why JSON is used instead.
A more sensible approach is parsing the xml on arrival, and store the data on a javascript object, that is accessed instead.

Most of the changes done on the template were adding a <span> tag with an id around the values that changed; a variation of this were the bars, where the change value, a percentage, was derived from the values received; another was the server connection status icon, which needed some more logic, and a reference in a hash of the image files.

The only hard case happens with the server connection status text: if connected to a server, the name of the server appears; if not , a translated string of the state appears; also, if disconnected, a link with translated text appears to connect to any server, not otherwise; both are together on the template.

The translation string problem was not resolved in the prototype; several solutions appear:

- a external javascript file with a translation table (pratical, but duplicates info and efforts, since such a table already exists in the main program);

- a translation table added runtime to the webpage, like all the changes in the template

- the translated strings in the XML response (not very pratical, since adds data that doesnt change to every response, only relevant to the webserver, and therefore useless to other clients).

The link problem was resolved in this fashion:

- separated the link from the connection status text on the template; this will affect the XP template, and possibly the Light one too (not checked).

-the link (and the brackets) was put inside a <span> tag with a display css attribute; by modifying the value of such attribute, its possible to hide the link when not disconnected, and show it when in such state.

This post has been edited by Darklord: May 23 2006, 19:50

[Darklord]

Here is the code for the prototype i have been working on

When I try to open the zip file (windows integrated ZIP support) I'm being told the file is corrupt..

Corrected, problem with the ftp client.

[Darklord]

Implementation Notes, part 5

Next Steps:

- refactoring of the webserver

- add login with response in XML ( needs to give back the session id )

- implement responses in XML for commands, so the clients can get feedback.

- define the format of the XML, maybe do a DTD.

- decide on the translation problem

- add all the other requests needed

- agree on a ajax framework to use

- rework the code using the framework, and add all the other code needed

- change the template, and update the rest.

[Darklord]

archive with executable

Edit: this may not have worked, since this version uses logging, and i didnt add the library to the archive. Corrected, follow the instructions on the README.

This post has been edited by Darklord: May 23 2006, 22:42

[Beltxo]

Hmmm... such an interesting development, implementation, anotation and so!!

Impresive!!

Keep working on it!!!

[Fuxie - DK]

archive with executable

Which source codes is this built on? Maybe you could change the version-string (in upcode.h I think) to reflect which code case you're using and indicate it's an Ajax build..

From my first test, it seems that this Ajax version is MUCH slower than before, but I will make some timings later, as I can be wrong..

Also, I can't use View -> Source.. Has that been disabled?

Issues

• Are you using old files?
  

  • [GZIP Enabled] [[Valid] CSS] [[Valid] HTML]

    
    

  • [GZIP Enabled] [Valid CSS] [Valid HTML]

[Fuxie - DK]

Test run, to check times...
Scenario:
Login -> Press Transfer -> Wait for page to display -> Press Preferences (to get last timestamp).

24-05-2006 08:11:18: WebServer: start request processing (Cmd=)
24-05-2006 08:11:18: WebServer: request prepared (Sz=1796)
24-05-2006 08:11:18: WebServer: request sent
24-05-2006 08:11:25: WebServer: start request processing (Cmd=password)
24-05-2006 08:11:25: WebServer: request prepared (Sz=5981)
24-05-2006 08:11:25: WebServer: request sent
24-05-2006 08:11:36: WebServer: start request processing (Cmd=transfer)
24-05-2006 08:17:59: WebServer: request prepared (Sz=192002) <-- 383 seconds to prepare page in eMule Plus
24-05-2006 08:17:59: WebServer: request sent
24-05-2006 08:20:20: WebServer: start request processing (Cmd=options) <-- 141 seconds to display page in MSIE. When pressed, WS timed out to Login Screen (Large download list)
24-05-2006 08:20:20: WebServer: request prepared (Sz=1796)
24-05-2006 08:20:20: WebServer: request sent

24-05-2006 08:33:59: WebServer: start request processing (Cmd=)
24-05-2006 08:33:59: WebServer: request prepared (Sz=1805)
24-05-2006 08:33:59: WebServer: request sent
24-05-2006 08:34:06: WebServer: start request processing (Cmd=password)
24-05-2006 08:34:06: WebServer: request prepared (Sz=6008)
24-05-2006 08:34:06: WebServer: request sent
24-05-2006 08:34:11: WebServer: start request processing (Cmd=transfer)
24-05-2006 08:40:25: WebServer: request prepared (Sz=191039) <-- 374 seconds to prepare page in eMule Plus
24-05-2006 08:40:25: WebServer: request sent
24-05-2006 08:42:38: WebServer: start request processing (Cmd=options)<-- 133 seconds to display page in MSIE. When pressed, WS timed out to Login Screen (Large download list)
24-05-2006 08:42:38: WebServer: request prepared (Sz=1805)
24-05-2006 08:42:38: WebServer: request sent

The numbers speak for them selves.. Ajax is only slightly slower than XP. IMHO, test-results are with the range of "random delays" such as a flush of diskcache, swapping memory to/from pagefile.sys etc.

[Darklord]

archive with executable

Which source codes is this built on? Maybe you could change the version-string (in upcode.h I think) to reflect which code case you're using and indicate it's an Ajax build..

From my first test, it seems that this Ajax version is MUCH slower than before, but I will make some timings later, as I can be wrong..

On Plus CVS, with the code in the cvs freeze (Sourceforge issue).

Also, I can't use View -> Source.. Has that been disabled?

No, of course not. (i dont believe that is even possible).

Issues

• Are you using old files?
  
  
  

  • [GZIP Enabled] [[Valid] CSS] [[Valid] HTML]

    
    
    

  • [GZIP Enabled] [Valid CSS] [Valid HTML]

    
    
  
  

Yes, see above.

[Darklord]

Test run, to check times...
Scenario:
Login -> Press Transfer -> Wait for page to display -> Press Preferences (to get last timestamp).

24-05-2006 08:11:18: WebServer: start request processing (Cmd=)
24-05-2006 08:11:18: WebServer: request prepared (Sz=1796)
24-05-2006 08:11:18: WebServer: request sent
24-05-2006 08:11:25: WebServer: start request processing (Cmd=password)
24-05-2006 08:11:25: WebServer: request prepared (Sz=5981)
24-05-2006 08:11:25: WebServer: request sent
24-05-2006 08:11:36: WebServer: start request processing (Cmd=transfer)
24-05-2006 08:17:59: WebServer: request prepared (Sz=192002) <-- 383 seconds to prepare page in eMule Plus
24-05-2006 08:17:59: WebServer: request sent
24-05-2006 08:20:20: WebServer: start request processing (Cmd=options) <-- 141 seconds to display page in MSIE. When pressed, WS timed out to Login Screen (Large download list)
24-05-2006 08:20:20: WebServer: request prepared (Sz=1796)
24-05-2006 08:20:20: WebServer: request sent

24-05-2006 08:33:59: WebServer: start request processing (Cmd=)
24-05-2006 08:33:59: WebServer: request prepared (Sz=1805)
24-05-2006 08:33:59: WebServer: request sent
24-05-2006 08:34:06: WebServer: start request processing (Cmd=password)
24-05-2006 08:34:06: WebServer: request prepared (Sz=6008)
24-05-2006 08:34:06: WebServer: request sent
24-05-2006 08:34:11: WebServer: start request processing (Cmd=transfer)
24-05-2006 08:40:25: WebServer: request prepared (Sz=191039) <-- 374 seconds to prepare page in eMule Plus
24-05-2006 08:40:25: WebServer: request sent
24-05-2006 08:42:38: WebServer: start request processing (Cmd=options)<-- 133 seconds to display page in MSIE. When pressed, WS timed out to Login Screen (Large download list)
24-05-2006 08:42:38: WebServer: request prepared (Sz=1805)
24-05-2006 08:42:38: WebServer: request sent

The numbers speak for them selves.. Ajax is only slightly slower than XP. IMHO, test-results are with the range of "random delays" such as a flush of diskcache, swapping memory to/from pagefile.sys etc.

That scenario is somewhat flawed.

Its possible that the first request may be a little longer due to a bigger template and getting the javascript file (if not on cache already), although 11 sec is a bit too much.

However, ajax works its magic AFTER the first load; subsequent updates should be much faster and smaller, due to the fact that the page its not sent complete, but just the xml request that can be obtained by the url given above. Also, the browser doesnt have to render the complete page, but just parts of it (the ones that change).

A better test scenario, to just check the changed parts would be to measure the time of the the first send, after the submit of the login, that only gives the stats bar.

Did you compare only templates? Or clients as well? And how did you measured the time?

I will look into this, and report the numbers i come with.

PS: the latest update done from cvs yesterday is broken and wont compile, something about the fake check.

This post has been edited by Darklord: May 24 2006, 11:48

[Fuxie - DK]

That scenario is somewhat flawed.

This is how we've always tested when any speed-improvements have been made. If you can some up with a better way, feel free to explain..

Actually, I had already "played" a little with the Ajax-template before I began to time it.. So we're already in "ajax works its magic AFTER the first load" process, and still 11/8 seconds slower than XP in first run.

When I tested Ajax-template, I use the Ajax-EXE file. When I tested XP, I used Nightly-20060523-1000

I timed it simply by using the timestamp (visible in the above examples) in DebugLog..

PS: the latest update done from cvs yesterday is broken and wont compile, something about the fake check.

That's strange, because our Nightlys are compiled twice a day without problems..