Hello All,
I will be posting all my updates and weekly status as reply to this post.
I am attaching my GSoC proposal.
Following is my Github repository :
https://github.com/ChokshiUtsav/BitTorrent
Currently following milestones have been reached :
1 : Decoding bencoded torrent file and extracting meta info : Partially Tested
2 : Generating HTTP req and Parsing HTTP response : Partially Tested
3 : Establishing connection with one of peers : Partially implemented.
Thanks,
Utsav Chokshi
IRC Name : Utsav_Chokshi
Developer Chat ID : Utsav_Chokshi
GSoC : BitTorrent
-
- Attachments
-
-
Utsav_Chokshi_GSoCProjectProposal_Final.pdf (153.66 KiB)Downloaded 683 times
-
Hello All,
This is my weekly report :
https://docs.google.com/document/d/1bMl ... sp=sharing
And I have some queries and I will post it here soon.
This is my weekly report :
https://docs.google.com/document/d/1bMl ... sp=sharing
And I have some queries and I will post it here soon.
Dunkaist sent me a patch for his torrent code the other day, related to the fact that port numbers for socket functions must be in network byte order, like on most other platforms. Is/was your problem perhaps related to this?Utsav_Chokshi wrote:I am not able to connect with peers. [check peer.asm]
I have verified my code by connecting to well-known IPs (telehack.com) and it works. Somehow I feel that there is problem with my net configuration.
Easy: Dont draw a GUI, if you want to hide from taskbar the firstUtsav_Chokshi wrote: I don’t know how to make a daemon/service in kolibrios which makes code to run in background. Any pointers ?
This one is trickier, the controls you have sketched do not really exist yet for kolibriOS.Utsav_Chokshi wrote: Good examples for building UI.
"Any intelligent fool can make things bigger, more complex, and more violent. It takes a touch of genius -- and a lot of courage -- to move in the opposite direction." Albert Einstein
Utsav_Chokshi
Unfortunately, the Widget toolkit task was not chosen this GSoC, but if you will write some GUI element as an external module, we will appreciate that. You also may try to cooperate with nisargshah95 about this.the controls you have sketched do not really exist yet for kolibriOS
@Pathoswithin ..
And yes I am in touch with NisargShah95.
@hidnplayr..
Thanks for answers.
But I did not understand sentence : "the first name of the program should be the @ sign" ?
As per discussuion with dunkaist,
For communication between torrent client UI and background program/daemon, using network socket seems preferable. As UI and daemon can be run on two different kolibrios boxes. Your suggestion and inputs are welcome.
Yes, It was problem related byte order.
I have made some other progress as well.
I'll update this thread along with some design quesions.
Thanks.
My GUI includes few buttons, open dialog, scroll bars and edit boxes. And I am using boxlib and proclib for that. I found it sufficient. Only control, I have not tried to incorporate yet is ScrollBar.Unfortunately, the Widget toolkit task was not chosen this GSoC, but if you will write some GUI element as an external module, we will appreciate that. You also may try to cooperate with nisargshah95 about this.
And yes I am in touch with NisargShah95.
@hidnplayr..
Thanks for answers.
I got the idea of not drawing GUI for program to run in background.Easy: Dont draw a GUI, if you want to hide from taskbar the first name of the program should be the @ sign, use IPC or local sockets to communicate with the program(s).
But I did not understand sentence : "the first name of the program should be the @ sign" ?
As per discussuion with dunkaist,
For communication between torrent client UI and background program/daemon, using network socket seems preferable. As UI and daemon can be run on two different kolibrios boxes. Your suggestion and inputs are welcome.
Yes I have received patch from dunkaist and I am now successfully able to communicate[handshake] with peers.Dunkaist sent me a patch for his torrent code the other day, related to the fact that port numbers for socket functions must be in network byte order, like on most other platforms. Is/was your problem perhaps related to this?
Yes, It was problem related byte order.
I have made some other progress as well.
I'll update this thread along with some design quesions.
Thanks.
See my attached example using shared mem functions.Utsav_Chokshi wrote: As per discussuion with dunkaist,
For communication between torrent client UI and background program/daemon, using network socket seems preferable. As UI and daemon can be run on two different kolibrios boxes. Your suggestion and inputs are welcome.
Its c-language, but using asm is easy too. SysFn 68.22
*the first char of the name should be @
Hello,
Following is over-all torrent client design I can think of :
[img] [/img]
Explanation :
FrontEnd.asm : GUI of application and consumes one thread
BackEnd.asm : Daemon/Backend of torrent application and consumes two thread
FronEnd and BackEnd communicates through network sockets over TCP protocol via server[backend]-client[frontend] mechanism.
TCP Protocol payload format [from frontend -> backend] is :
<LengthOfMessage><TypeOfMessage><PayloadOfMessage> [4Bytes-1Byte-Nbytes]
Type of Message can be :
TORRENT_ADD(1) (Payload : Torrent_ID : Torrent File Path : Download Location)
TORRENT_START(2) (Payload : Torrent_ID)
TORRENT_PAUSE(3) (Payload : Torrent_ID)
TORRENT_REMOVE(4) (Payload : Torrent_ID)
TORRENT_SHOW(5) (Payload : Torrent_ID)
TORRENT_QUIT(6) (This message closes backend)
TCP Protocol Payload format for messages[ from backend -> frontend ] is :
<LengthOfMessage><PayloadOfMessage>
In frontend:
There is only one thread which draws UI as well as connects with backend and updates it.
In backend :
Thread one handles all incoming request from frontend and provides neccessary information.
Thread two handles all torrents' downlaoding and uploading.
These two threads of backend share one common data-area [/list] : List of Torrents.
Thread one mainly consumes this list and add/remove torrents
Thread two mainly updates this list.
Each element of list follows torrent structure described in torrent.inc [Few entries like torrent_id , is yet to be made]
-> Thread two loops over all torrents in torrent list.
-> For each torrent, it works for every torrernt for around 5000ms. [this time is subject to experiment and yet to optimize].
-> For each torrent, it connects with tracker server [if timeout is happended for connecting with tracker server , which is usually 1800 secs (30 mins)]
it connects with all active peers, offering pieces to download.
it sends keep-alive messages to inactive peers [2 minute is max ideal period] or close connection with them.
it accepts incoming connections from other peers [at max 16 peers needs to be handled , according to torrent guideline ] and uploads pieces to them.
-> Uploading and downloading starts at two timeout events within 5000 ms span.
Credits :
-> Idea of using network socket over IPC mechanism is conclusion of discussion with Dunkaist.
Benefit of using network socket over IPC , is , that we can have backend and frontend on two different kolibrios boxes.
-> Keeping thread numbers as minimal as possible is conclusion of discussion with Hidnplayr.
As kolibrios , as a whole , supports only 256 threads.
-> Keeping timeouts for handling multiple connections is also concluson of discussion with Hidnplayr.
So that fairness can be ensured.
Help required :
-> Currently I am not able to connect backend with frontend using localhost as IP [127.0.0.1]. My program gets stuck at connect system call with interrupt 40h [provided server is up]. Any help ?
-> I have attached sample tcp-server and tcp-client programs for the same.
-> Currently, I am able to open file browser dialog and copies a path of selected file to editbox.
-> Problem, I am facing is : I am not able to select folder. . I guess it has something to do with filter of open dialog.
Current Progress :
-> I am able to download one piece-block from peer.
-> Basic structure of GUI is almost ready and I am much clear regarding the over-all design.
Github Link : https://github.com/chokshiutsav/bittorrent
Suggestion regarding any desing aspect is welcome.
Thanks,
Utsav Chokshi
Following is over-all torrent client design I can think of :
[img] [/img]
Explanation :
FrontEnd.asm : GUI of application and consumes one thread
BackEnd.asm : Daemon/Backend of torrent application and consumes two thread
FronEnd and BackEnd communicates through network sockets over TCP protocol via server[backend]-client[frontend] mechanism.
TCP Protocol payload format [from frontend -> backend] is :
<LengthOfMessage><TypeOfMessage><PayloadOfMessage> [4Bytes-1Byte-Nbytes]
Type of Message can be :
TORRENT_ADD(1) (Payload : Torrent_ID : Torrent File Path : Download Location)
TORRENT_START(2) (Payload : Torrent_ID)
TORRENT_PAUSE(3) (Payload : Torrent_ID)
TORRENT_REMOVE(4) (Payload : Torrent_ID)
TORRENT_SHOW(5) (Payload : Torrent_ID)
TORRENT_QUIT(6) (This message closes backend)
TCP Protocol Payload format for messages[ from backend -> frontend ] is :
<LengthOfMessage><PayloadOfMessage>
In frontend:
There is only one thread which draws UI as well as connects with backend and updates it.
In backend :
Thread one handles all incoming request from frontend and provides neccessary information.
Thread two handles all torrents' downlaoding and uploading.
These two threads of backend share one common data-area [/list] : List of Torrents.
Thread one mainly consumes this list and add/remove torrents
Thread two mainly updates this list.
Each element of list follows torrent structure described in torrent.inc [Few entries like torrent_id , is yet to be made]
-> Thread two loops over all torrents in torrent list.
-> For each torrent, it works for every torrernt for around 5000ms. [this time is subject to experiment and yet to optimize].
-> For each torrent, it connects with tracker server [if timeout is happended for connecting with tracker server , which is usually 1800 secs (30 mins)]
it connects with all active peers, offering pieces to download.
it sends keep-alive messages to inactive peers [2 minute is max ideal period] or close connection with them.
it accepts incoming connections from other peers [at max 16 peers needs to be handled , according to torrent guideline ] and uploads pieces to them.
-> Uploading and downloading starts at two timeout events within 5000 ms span.
Credits :
-> Idea of using network socket over IPC mechanism is conclusion of discussion with Dunkaist.
Benefit of using network socket over IPC , is , that we can have backend and frontend on two different kolibrios boxes.
-> Keeping thread numbers as minimal as possible is conclusion of discussion with Hidnplayr.
As kolibrios , as a whole , supports only 256 threads.
-> Keeping timeouts for handling multiple connections is also concluson of discussion with Hidnplayr.
So that fairness can be ensured.
Help required :
-> Currently I am not able to connect backend with frontend using localhost as IP [127.0.0.1]. My program gets stuck at connect system call with interrupt 40h [provided server is up]. Any help ?
-> I have attached sample tcp-server and tcp-client programs for the same.
-> Currently, I am able to open file browser dialog and copies a path of selected file to editbox.
-> Problem, I am facing is : I am not able to select folder. . I guess it has something to do with filter of open dialog.
Current Progress :
-> I am able to download one piece-block from peer.
-> Basic structure of GUI is almost ready and I am much clear regarding the over-all design.
Github Link : https://github.com/chokshiutsav/bittorrent
Suggestion regarding any desing aspect is welcome.
Thanks,
Utsav Chokshi
Subject : Regarding writing pieces of file being downloaded and merging them in a single file
As per discussion with ashmew2 @IRC,
- A piece of torrent file can be as large as 512-1024kB , which in turn gets downloaded as blocks of 4kBs.
- So strategy I decided is , downloading blocks of single piece from single peer in a sequence .
- Keep Writing all those blocks in separate piece-file.
- Once, the complete piece is downloaded. It is verified against its Hash. [P.S. Hash of each piece is stored in ".torrent" file]
- We need to write that piece to bigger file [Or you can say merge with other pieces].
- For that, a file (call this "File X") equal to original file-size needs to be created before download starts.
- As length of each piece is same and known before-hand and index of piece is also known , we can write piece to File X at specifc
offset [(index-1)*(piece-length)].
Here I am attaching demo program which replicates last two steps. Explanation for demo program :
It creates file of size 100kb using Function 70 , subfunction 2 and 4.
It writes 8 byte of data at offset 128 and offset 256 with respect to start position (i.e. byte 0)
@ashmew2...Function 70, subfunction 4 writes 0 to all extended bytes by default. [With reference to our discussion of initialzing file with zero]
~Utsav Chokshi
As per discussion with ashmew2 @IRC,
- A piece of torrent file can be as large as 512-1024kB , which in turn gets downloaded as blocks of 4kBs.
- So strategy I decided is , downloading blocks of single piece from single peer in a sequence .
- Keep Writing all those blocks in separate piece-file.
- Once, the complete piece is downloaded. It is verified against its Hash. [P.S. Hash of each piece is stored in ".torrent" file]
- We need to write that piece to bigger file [Or you can say merge with other pieces].
- For that, a file (call this "File X") equal to original file-size needs to be created before download starts.
- As length of each piece is same and known before-hand and index of piece is also known , we can write piece to File X at specifc
offset [(index-1)*(piece-length)].
Here I am attaching demo program which replicates last two steps. Explanation for demo program :
It creates file of size 100kb using Function 70 , subfunction 2 and 4.
It writes 8 byte of data at offset 128 and offset 256 with respect to start position (i.e. byte 0)
@ashmew2...Function 70, subfunction 4 writes 0 to all extended bytes by default. [With reference to our discussion of initialzing file with zero]
~Utsav Chokshi
Note, that for big files optimal data block size for sysfunction 70.3 is 16 Mb.
Also note, that currently, for performance reasons, sysfunction 70.4 on NTFS erases up to 16 Mb, and just extends if operation is bigger (and not tested yet).
Also note, that currently, for performance reasons, sysfunction 70.4 on NTFS erases up to 16 Mb, and just extends if operation is bigger (and not tested yet).
What do you exactly mean by data block size ?Pathoswithin wrote:Note, that for big files optimal data block size for sysfunction 70.3 is 16 Mb.
Do you mean, allocating file-size in multiples of 16Mb ?
I mean writing to allocated file in multiples of 16Mb. Hard drive spins and has big delay at operation start.
Hello All,
Subject : Single Piece Download and Upload Strategy and Overview of file operations
Before Bittorrent Client starts contacting peers :
1) Allocate file-space :
[In case of multi-file torrent]
1) Create directory. Name of directory is provided in .torrent file and stored in torrent structure.
2) Create empty files of size equal to files to be downloaded.
[In case of single-file torrent]
1) Create empty file of size equal to file to be downloaded.
1) Fill following piece structure :
An array of piece structure is created and filled.
Please note that,
-> Few member variables , like index and download_status , may be redundant.
As piece structure is stored in form of an array , so index may not be needed to store.
As bitfiled of overall torrent is stored, download_status may not be needed to store.
-> Array of piece structure will be kept in memory.
How much is memory required ? => For 2GB of torrent file roughly 300-500kB is required.
Download a piece :
1) Download all blocks of piece from single peer.
2) Kepp all blocks of piece in memory till complete piece is downloaded [It requires 512kB (standard piece size) memory]
3) Create SHA1 hash of piece [createSHA1Hash()]
4) getHashOfPiece()
5) VerifyHashOfPiece()
6) setpiece(torrent,piece_index) [it wirtes piece to appropriate files]
setpiece(torrent,piece_index)
-> jump to (piece_index-1)*(piece structure size)
-> loop over i ( i : 1 to num_files )
-> open file for write
-> seek file to file_offset
-> skeip piece_offset bytes in piece
-> mov bytes from piece to file for length
-> close file
-> delete piece
Upload a piece :
1) getpiece(torrent, piece_index) [it brings piece data from different files to memory]
2) upload all blocks to peer , one by one.
getpiece(torrent, piece_index)
-> Allocate memory for piece
-> jump to (piece_index-1)*(piece structure size)
-> loop over i ( i : 1 to num_files )
-> open file for read
-> seek file to file_offset
-> skip piece_offset bytes in memory allocated for piece
-> mov bytes from file to piece for length
-> close file
For future reference :
[i]Generally, today's bittorrent client provides facility of downloading a single file (or set of selected files) out of complete torrent.
This problem drills down to download set of pieces that belongs to that file.
How to know which set of pieces to download for particular set of file(s) ?
Possible Solution :
Assign each file an index (starting from 1 to num_of_files) and store it within piece structure.
Now, We can apply binary search over piece structure array and look which file index it contains.
If file index is lesser or equal, search in left.
Otherwise, search in right.
Continue till you find piece that contains that file-index with corresponding file-offset 0.
Credits :
These are fruits of discussing with Dunkaist.
@pathoswithin..
If it is the case, I doubt that I would be able to do that, as I will be writing one piece [512kB] to hard drive [may be to different files], due to strict memory requirements.
@dunkaist... Thanks,
Utsav Chokshi
Subject : Single Piece Download and Upload Strategy and Overview of file operations
Before Bittorrent Client starts contacting peers :
1) Allocate file-space :
[In case of multi-file torrent]
1) Create directory. Name of directory is provided in .torrent file and stored in torrent structure.
2) Create empty files of size equal to files to be downloaded.
[In case of single-file torrent]
1) Create empty file of size equal to file to be downloaded.
1) Fill following piece structure :
An array of piece structure is created and filled.
Code: Select all
struct piece
index dd ?
piece_hash rb 20
num_files dd ? ;array_size and it defines number of files' data contained by this piece
piece_offset dd ? ;array
length dd ? ;array
file_offset dd ? ;array
file_names dd ? ;array
num_blocks_downloaded dd ?
download_status dd ?
ends
BT_PIECE_DOWNLOAD_NOT_STARTED = 0
BT_PIECE_DOWNLOAD_IN_PROGRESS = 1
BT_PIECE_DOWNLOAD_COMPLETE = 2
-> Few member variables , like index and download_status , may be redundant.
As piece structure is stored in form of an array , so index may not be needed to store.
As bitfiled of overall torrent is stored, download_status may not be needed to store.
-> Array of piece structure will be kept in memory.
How much is memory required ? => For 2GB of torrent file roughly 300-500kB is required.
Download a piece :
1) Download all blocks of piece from single peer.
2) Kepp all blocks of piece in memory till complete piece is downloaded [It requires 512kB (standard piece size) memory]
3) Create SHA1 hash of piece [createSHA1Hash()]
4) getHashOfPiece()
5) VerifyHashOfPiece()
6) setpiece(torrent,piece_index) [it wirtes piece to appropriate files]
setpiece(torrent,piece_index)
-> jump to (piece_index-1)*(piece structure size)
-> loop over i ( i : 1 to num_files )
-> open file for write
-> seek file to file_offset
-> skeip piece_offset bytes in piece
-> mov bytes from piece to file for length
-> close file
-> delete piece
Upload a piece :
1) getpiece(torrent, piece_index) [it brings piece data from different files to memory]
2) upload all blocks to peer , one by one.
getpiece(torrent, piece_index)
-> Allocate memory for piece
-> jump to (piece_index-1)*(piece structure size)
-> loop over i ( i : 1 to num_files )
-> open file for read
-> seek file to file_offset
-> skip piece_offset bytes in memory allocated for piece
-> mov bytes from file to piece for length
-> close file
For future reference :
[i]Generally, today's bittorrent client provides facility of downloading a single file (or set of selected files) out of complete torrent.
This problem drills down to download set of pieces that belongs to that file.
How to know which set of pieces to download for particular set of file(s) ?
Possible Solution :
Assign each file an index (starting from 1 to num_of_files) and store it within piece structure.
Now, We can apply binary search over piece structure array and look which file index it contains.
If file index is lesser or equal, search in left.
Otherwise, search in right.
Continue till you find piece that contains that file-index with corresponding file-offset 0.
Credits :
These are fruits of discussing with Dunkaist.
@pathoswithin..
Is 16Mb -> 16 Megabits -> 2000 kilo Bytes ?I mean writing to allocated file in multiples of 16Mb. Hard drive spins and has big delay at operation start.
If it is the case, I doubt that I would be able to do that, as I will be writing one piece [512kB] to hard drive [may be to different files], due to strict memory requirements.
@dunkaist... Thanks,
Utsav Chokshi
Which requirements?I will be writing one piece [512kB] to hard drive [may be to different files], due to strict memory requirements
Mentors
I see, you have ignored my plea to communicate with student publicly.
IMHO, this is something that should be configurable, as in any decent client.Pathoswithin wrote:Which requirements?I will be writing one piece [512kB] to hard drive [may be to different files], due to strict memory requirements
"Any intelligent fool can make things bigger, more complex, and more violent. It takes a touch of genius -- and a lot of courage -- to move in the opposite direction." Albert Einstein
Who is online
Users browsing this forum: No registered users and 2 guests