Outline · [ Standard ] · Linear+ Ranomization of Parts, Increased chance of completing downloads

[Mystif]

Is it possible to introduce some kind of sudo-randomization to the part request? While it does not happen all the time I am occationally frustrated by stalled downloads.

What happens is that there will be one or two sources for a file and 20 people - all downloading nearly the exact same parts. When the one or two sources go offline the rest have nothing beyond a few parts to exchange with each other.

Right now I am looking at file.iso. There are six of us online and we are all missing the exact same (roughly) 13 parts. We are all stuck unless the one and only source for the file returns online. There are about 87 parts to this file and I received less than three from sources other than the original poster of the file.

If each of our eMule Plus clients chose a different order based on sudo-randomization, rather than the current rules, then chances are we would be able to exchange the remaining parts amoungst ourselves.

Ultimately, if I am DLing 2, someone else 27, and another person 13 then we each have something someone else can use if the primary source goes offline. If we each DL 2 then when the primary sorce goes offline we all wait, and wait, and wait.

(I did look and did not see a request that looked like this one, to me. If someone has already requested this feature then I am sorry for the duplication.)

This post has been edited by reanimated838uk: Nov 13 2005, 13:22

[muleteer]

1. No filenames please.

2. This is already done. The client automatically requests the least distributed part (rarest) so that you can become an additional source for that part.

3. It is happening the way you propose. However, you can only upload a part if you have the whole part, so if you only managed to get a little bit of data in the part, you can't upload it.

4. It seems weird because the original releaser is a very slow uploader. Most of your data must have come from other clients; anyone who gets a part from the original/complete source quickly uploads it to the others, so you soon end up with exactly the same parts.

5. Try PMing the complete source and ask him to put the file on release priority (in his shared files). That should speed things up.

[Mystif]

Thank you for responding so fast, muleteer.

I wasn't concerned about the filename because it was the author's choice to distribute the file this way. (DL from the official web site for a fee, or DL from P2P for free - free fits my budget. ) I can understand the concern over filenames, I will respect that rule in the future.

This file is (as far as I know) the only file that this person is distributing. It also seems that the distribution times for the file are restricted to weekdays. Seems this person turns off the computer when they leave work.

The DL process has never looked random or "rarest first" to me - no offense intended.

Random:
Is a tough call. It is possible that eveyone ends up DLing the exact same part randomly.

Rarest:
I join a "feeding frenzy" already in progress. There are at least 50 other clients DLing from each other and as many as ten clients with the complete file. I will get various parts from various clients as I move through the queues... but when I get to the front of the queue of a client with the complete file I get part 47. I have to wonder why, afterall, five others already have 47 but not one single client (other than the ones with the complete file) have part 73. Why didn't my client request 73?

This is not a complaint, it is a curiosity. If I did not like the program I would not use it.

[Aw3]

I have to wonder why, afterall, five others already have 47 but not one single client (other than the ones with the complete file) have part 73.  Why didn't my client request 73?

Because your client already had downloaded something from part 47. In this case it tries to finalize it.

[muleteer]

Hmm..the rarest part requesting algorithm is overridden by two things: 'get first/ last chunks for preview' and when a part is already partially filled (the intention here being to complete the part as quickly as possible so you can become an uploader).

Most of the time this works perfectly, but maybe we need to take another look at the scenario you have described. Maybe by detecting the number of complete sources... We are already doing something similar for rare files; maybe we could apply that to just released files also.

[Mystif]

First an update about the availability of the complete file - There is a complete copy available right now, first time I have seen it this weekend, but I guess that means it is not weekdays only like I had come to believe.

To Aw3;
Good point, but only to a point. Concider that at the moment I know that part 50 is at 75 percent. I understand that my client will finish that part before moving on to the next one. But the root of my curiosity is in wondering why six other clients are all waiting to finish either that part or the one my client DLed before it. Not one of the seven of these clients is attempting to DL the parts in a different order. So to me there seems to be no indication of randomization or rarest, rather all the clients seem to draw the same conclusions about which part to DL next.

Parts 1, 8, 15, 34, 44, etc... have all been ignored by all of the clients up to now.

I would think that if rarest counts then once several clients get part 50 the remaining clients would no longer consider it to be rarest and would choose a different part. Instead all seven clients have either chosen, or will choose part 50 before moving on to the next part. (Six clients, incuding mine are on part 50, one client is on part 22 - the part my client last completed.)

To muleteer;
It is not an A/V file so "get first/last..." does not apply. As to partial parts: I discussed that above.

I guess what I imagine is that if it was rarest and sudo-random something like the following would occur:

1) My client sees that there are six other clients with missing parts, and one with the complete file.

2) When my client reaches the front of the queue on any other client it "asks" the following:

a. Is this a client with the complete file? If so, choose the part to be DLed by selecting a random part from the list of parts which no other client has DLed yet.

b. If no, does this client have any parts that qualify as "rarest"? ie: parts that only this client and the clients with complete files have?

1. If yes, choose randomly from those parts only.

2. If no, choose the most rare part. One that has only been DLed by two clients and so on...

The end result of this would be that when there are no clients with complete sources nothing much would change - eventually every client will have the same parts. But when there are complete sources avilable and clients access them each client gets a different part, then if the client(s) with the complete source(s) does/do go offline there will be more parts available to exchange. This would make the effort much more co-operative.

[muleteer]

This is exactly how things are working right now. Where things are getting screwed up is that the complete source stops transferring before a part is completed. In your example, you started downloading part 50, but got kicked out after downloading, say, 5MB. Another client got it's turn, saw that no one else had part 50, and started downloading it, but it too got disconnected after, say, 4 MB....and so on. Until someone completes part 50, every client will try to get the same part because that's the rarest one. No one knows how much or how little data has been received for a part by another client. The source is supposed to go on transferring until the current part is complete before moving on to the next waiting client, but it seems that this is not taking place.

Now when your turn comes again, your client will again try to complete part 50, the one for which it has already received some data, and as per current logic, it will request this part even if another client (besides the complete source) has already got this part. This is where I can see possible scope for improvement. We could try changing this so that the client only requests parts from the complete source that no one else has got. Of course, then you will moan about clients that take one part, but don't upload it to others as efficiently as you would like.

[Aw3]

I would think that if rarest counts then once several clients get part 50 the remaining clients would no longer consider it to be rarest and would choose a different part.  Instead all seven clients have either chosen, or will choose part 50 before moving on to the next part.  (Six clients, incuding mine are on part 50, one client is on part 22 - the part my client last completed.)

Remote client just don't see a part downloaded by you for a while (up to 1 hour).
I don't know where you got the information which remote client is downloading which part from some remote source...
And besides that here we talked about eMule Plus which is only one client, there are also eMule, etc. They use different algorithms to select a chunk.

    b. If no, does this client have any parts that qualify as "rarest"? ie: parts that only this client and the clients with complete files have?

      1. If yes, choose randomly from those parts only.

      2. If no, choose the most rare part. One that has only been DLed by two clients      and so on...

This is how it works right now.

[Mystif]

To Aw3 ;

"... I don't know where you got the information which remote client is downloading which part from some remote source...."

I simply watch the part bars fill in on the other clients that I can see. It appears that we are consistantly going after the same parts, one right after the other. It cannot tell me which client a client is gettingthe file from but it does show a distinct pattern of like progress.

To muleteer;
Should I start complaining now, or should I wait until that change is implimented?

In reference to a previous statement of mine, you said, "In your example, you started downloading part 50, but got kicked out after downloading, say, 5MB. Another client got it's turn, saw that no one else had part 50..."

And right there is the heart of my question, why does the next client go after 50 since no-one has 50, or 1, or 8 for that matter. All three are "rarest" since only one client has all three. So why do all the clients decide that the same part is "rarest"? This is where I think sudo-randomization would be best applied. If various clients try for different parts then more clients have a wider variety of parts to offer.

And as to "We could try changing this so that the client only requests parts from the complete source that no one else has got." I guess, in a way, that is almost what I am asking about.

What happens now is that the clients (to continue the example) try to get 50, each in their own turn. If the complete source goes offline and one or more have managed to complete 50 then each of the others will likely catch up... but that's it. There is nothing else to do unless a complete source or varied source comes online.

But what if some clients managed to go after, and get, 1, some 8 and others 50? Then when the complete source goes offline a client may have only one choice per queue, but they have three parts available across the queues (1/8/50) instead of just one (50). To me - not an expert in these matters - it would seem that this would increase the chances of complete transfers.

Here is another way to look at it. Right now I am receiving one part from the complete source (72) and show "no needed parts" for all the other clients. We all have parts 1, 15, and 72 left. Some have more left to get because they got here later, but only the complete source has these three parts. If some other client got part 1 instead of 72 (which those of us needing only three parts are currently after) I would be on two queues soon, instead of staying on one - trying for both 1 and 72. If it did work this way then even if the complete source goes offline I can still get 1. Right now that is not possible.

To both of you;
Thank you for explaining as much as you have, and for listening to me as I struggle to understand. I use eMule Plus because overall I like it the best. That is why early on I stated that my questions were not meant as complaints. Please keep up the good work and try to understand that I would never even try to write a P2P client, I rely on people with talent like yours to do the things that I cannot.

[muleteer]

Hmm...any ideas on where this pseudo-randomization would be implemented? I mean at the downloading end or at the uploading end.

If the releasing client (the complete source) had set the file to JumpStart priority (only for releasers) things would have been much better. This very interesting feature hides parts that have aready been uploaded to one user from all the other users downloading the file. The other users therefore have no choice but to download other (not yet uploaded) parts from the complete source.

However, this is useful mostly when downloading from release sites because the feature does not allow the complete source to be seen as having all the parts, so people who don't know that the file is under JumpStart think that it is an incomplete file and cancel the download. If you get the link from a release site, you know that the file is under JumpStart and continue downloading.

Should I start complaining now, or should I wait until that change is implimented?

Sadly, this problem is all too real. Just look at your queues and see how many clients have uploaded either nothing or very little data to you in spite of having you in their queues for days. Now imagine if one of these clients had a part that no one except the complete source had.