GSoC : BitTorrent

...
  • Hello All,

    This is my weekly report :
    https://docs.google.com/document/d/1bMl ... sp=sharing

    And I have some queries and I will post it here soon.
  • Utsav_Chokshi wrote:I am not able to connect with peers. [check peer.asm]
    I have verified my code by connecting to well-known IPs (telehack.com) and it works. Somehow I feel that there is problem with my net configuration.
    Dunkaist sent me a patch for his torrent code the other day, related to the fact that port numbers for socket functions must be in network byte order, like on most other platforms. Is/was your problem perhaps related to this?
    Utsav_Chokshi wrote: I don’t know how to make a daemon/service in kolibrios which makes code to run in background. Any pointers ?
    Easy: Dont draw a GUI, if you want to hide from taskbar the first name character of the program name should be the @ sign, use IPC or local sockets to communicate with the program(s).
    Utsav_Chokshi wrote: Good examples for building UI.
    This one is trickier, the controls you have sketched do not really exist yet for kolibriOS.
    "Any intelligent fool can make things bigger, more complex, and more violent. It takes a touch of genius -- and a lot of courage -- to move in the opposite direction." Albert Einstein
  • Utsav_Chokshi
    the controls you have sketched do not really exist yet for kolibriOS
    Unfortunately, the Widget toolkit task was not chosen this GSoC, but if you will write some GUI element as an external module, we will appreciate that. You also may try to cooperate with nisargshah95 about this.
  • @Pathoswithin ..
    Unfortunately, the Widget toolkit task was not chosen this GSoC, but if you will write some GUI element as an external module, we will appreciate that. You also may try to cooperate with nisargshah95 about this.
    My GUI includes few buttons, open dialog, scroll bars and edit boxes. And I am using boxlib and proclib for that. I found it sufficient. Only control, I have not tried to incorporate yet is ScrollBar.
    And yes I am in touch with NisargShah95.

    @hidnplayr..
    Thanks for answers.
    Easy: Dont draw a GUI, if you want to hide from taskbar the first name of the program should be the @ sign, use IPC or local sockets to communicate with the program(s).
    I got the idea of not drawing GUI for program to run in background.
    But I did not understand sentence : "the first name of the program should be the @ sign" ?

    As per discussuion with dunkaist,
    For communication between torrent client UI and background program/daemon, using network socket seems preferable. As UI and daemon can be run on two different kolibrios boxes. Your suggestion and inputs are welcome.
    Dunkaist sent me a patch for his torrent code the other day, related to the fact that port numbers for socket functions must be in network byte order, like on most other platforms. Is/was your problem perhaps related to this?
    Yes I have received patch from dunkaist and I am now successfully able to communicate[handshake] with peers.
    Yes, It was problem related byte order.

    I have made some other progress as well.
    I'll update this thread along with some design quesions.

    Thanks.
  • Utsav_Chokshi wrote: As per discussuion with dunkaist,
    For communication between torrent client UI and background program/daemon, using network socket seems preferable. As UI and daemon can be run on two different kolibrios boxes. Your suggestion and inputs are welcome.
    See my attached example using shared mem functions.
    Its c-language, but using asm is easy too. SysFn 68.22
    Attachments
    shm_run.c (735 Bytes)
    Downloaded 380 times
    runner.c (1.01 KiB)
    Downloaded 373 times
  • *the first char of the name should be @
  • Hello,

    Following is over-all torrent client design I can think of :
    [img]
    bittorrent_design.jpg
    bittorrent_design.jpg (238.53 KiB)
    Viewed 12485 times
    [/img]

    Explanation :
    FrontEnd.asm : GUI of application and consumes one thread
    BackEnd.asm : Daemon/Backend of torrent application and consumes two thread

    FronEnd and BackEnd communicates through network sockets over TCP protocol via server[backend]-client[frontend] mechanism.
    TCP Protocol payload format [from frontend -> backend] is :
    <LengthOfMessage><TypeOfMessage><PayloadOfMessage> [4Bytes-1Byte-Nbytes]
    Type of Message can be :
    TORRENT_ADD(1) (Payload : Torrent_ID : Torrent File Path : Download Location)
    TORRENT_START(2) (Payload : Torrent_ID)
    TORRENT_PAUSE(3) (Payload : Torrent_ID)
    TORRENT_REMOVE(4) (Payload : Torrent_ID)
    TORRENT_SHOW(5) (Payload : Torrent_ID)
    TORRENT_QUIT(6) (This message closes backend)

    TCP Protocol Payload format for messages[ from backend -> frontend ] is :
    <LengthOfMessage><PayloadOfMessage>

    In frontend:
    There is only one thread which draws UI as well as connects with backend and updates it.

    In backend :
    Thread one handles all incoming request from frontend and provides neccessary information.
    Thread two handles all torrents' downlaoding and uploading.

    These two threads of backend share one common data-area [/list] : List of Torrents.
    Thread one mainly consumes this list and add/remove torrents
    Thread two mainly updates this list.
    Each element of list follows torrent structure described in torrent.inc [Few entries like torrent_id , is yet to be made]

    -> Thread two loops over all torrents in torrent list.
    -> For each torrent, it works for every torrernt for around 5000ms. [this time is subject to experiment and yet to optimize].
    -> For each torrent, it connects with tracker server [if timeout is happended for connecting with tracker server , which is usually 1800 secs (30 mins)]
    it connects with all active peers, offering pieces to download.
    it sends keep-alive messages to inactive peers [2 minute is max ideal period] or close connection with them.
    it accepts incoming connections from other peers [at max 16 peers needs to be handled , according to torrent guideline ] and uploads pieces to them.
    -> Uploading and downloading starts at two timeout events within 5000 ms span.

    Credits :
    -> Idea of using network socket over IPC mechanism is conclusion of discussion with Dunkaist.
    Benefit of using network socket over IPC , is , that we can have backend and frontend on two different kolibrios boxes.

    -> Keeping thread numbers as minimal as possible is conclusion of discussion with Hidnplayr.
    As kolibrios , as a whole , supports only 256 threads.

    -> Keeping timeouts for handling multiple connections is also concluson of discussion with Hidnplayr.
    So that fairness can be ensured.

    Help required :
    -> Currently I am not able to connect backend with frontend using localhost as IP [127.0.0.1]. My program gets stuck at connect system call with interrupt 40h [provided server is up]. Any help ?
    -> I have attached sample tcp-server and tcp-client programs for the same.
    tcp_client.asm (3.45 KiB)
    Downloaded 384 times
    tcp_server.asm (4.65 KiB)
    Downloaded 379 times
    -> Currently, I am able to open file browser dialog and copies a path of selected file to editbox.
    -> Problem, I am facing is : I am not able to select folder. . I guess it has something to do with filter of open dialog.

    Current Progress :
    -> I am able to download one piece-block from peer.
    -> Basic structure of GUI is almost ready and I am much clear regarding the over-all design.

    Github Link : https://github.com/chokshiutsav/bittorrent

    Suggestion regarding any desing aspect is welcome. :)

    Thanks,
    Utsav Chokshi
  • Subject : Regarding writing pieces of file being downloaded and merging them in a single file

    As per discussion with ashmew2 @IRC,

    - A piece of torrent file can be as large as 512-1024kB , which in turn gets downloaded as blocks of 4kBs.
    - So strategy I decided is , downloading blocks of single piece from single peer in a sequence .
    - Keep Writing all those blocks in separate piece-file.
    - Once, the complete piece is downloaded. It is verified against its Hash. [P.S. Hash of each piece is stored in ".torrent" file]
    - We need to write that piece to bigger file [Or you can say merge with other pieces].
    - For that, a file (call this "File X") equal to original file-size needs to be created before download starts.
    - As length of each piece is same and known before-hand and index of piece is also known , we can write piece to File X at specifc
    offset [(index-1)*(piece-length)].

    Here I am attaching demo program which replicates last two steps.
    Downloaded 376 times
    Explanation for demo program :
    It creates file of size 100kb using Function 70 , subfunction 2 and 4.
    It writes 8 byte of data at offset 128 and offset 256 with respect to start position (i.e. byte 0)

    @ashmew2...Function 70, subfunction 4 writes 0 to all extended bytes by default. :) [With reference to our discussion of initialzing file with zero]

    ~Utsav Chokshi
  • Note, that for big files optimal data block size for sysfunction 70.3 is 16 Mb.

    Also note, that currently, for performance reasons, sysfunction 70.4 on NTFS erases up to 16 Mb, and just extends if operation is bigger (and not tested yet).
  • Pathoswithin wrote:Note, that for big files optimal data block size for sysfunction 70.3 is 16 Mb.
    What do you exactly mean by data block size ?
    Do you mean, allocating file-size in multiples of 16Mb ?
  • I mean writing to allocated file in multiples of 16Mb. Hard drive spins and has big delay at operation start.
  • Hello All,

    Subject : Single Piece Download and Upload Strategy and Overview of file operations

    Before Bittorrent Client starts contacting peers :

    1) Allocate file-space :
    [In case of multi-file torrent]
    1) Create directory. Name of directory is provided in .torrent file and stored in torrent structure.
    2) Create empty files of size equal to files to be downloaded.
    [In case of single-file torrent]
    1) Create empty file of size equal to file to be downloaded.

    1) Fill following piece structure :
    An array of piece structure is created and filled.

    Code: Select all

    struct piece
            index                   	dd ?
            piece_hash                rb 20
            num_files	         	dd ?  ;array_size and it defines number of files' data contained by this piece
            piece_offset         	dd ?  ;array
            length                 	 dd ?  ;array
            file_offset             	 dd ?  ;array
            file_names             	 dd ?  ;array
            num_blocks_downloaded   dd ?
            download_status         dd ?
    ends
    
    BT_PIECE_DOWNLOAD_NOT_STARTED = 0
    BT_PIECE_DOWNLOAD_IN_PROGRESS = 1
    BT_PIECE_DOWNLOAD_COMPLETE    = 2
    
    Please note that,
    -> Few member variables , like index and download_status , may be redundant.
    As piece structure is stored in form of an array , so index may not be needed to store.
    As bitfiled of overall torrent is stored, download_status may not be needed to store.

    -> Array of piece structure will be kept in memory.
    How much is memory required ? => For 2GB of torrent file roughly 300-500kB is required.

    Download a piece :
    1) Download all blocks of piece from single peer.
    2) Kepp all blocks of piece in memory till complete piece is downloaded [It requires 512kB (standard piece size) memory]
    3) Create SHA1 hash of piece [createSHA1Hash()]
    4) getHashOfPiece()
    5) VerifyHashOfPiece()
    6) setpiece(torrent,piece_index) [it wirtes piece to appropriate files]

    setpiece(torrent,piece_index)
    -> jump to (piece_index-1)*(piece structure size)
    -> loop over i ( i : 1 to num_files )
    -> open file for write
    -> seek file to file_offset
    -> skeip piece_offset bytes in piece
    -> mov bytes from piece to file for length
    -> close file
    -> delete piece

    Upload a piece :
    1) getpiece(torrent, piece_index) [it brings piece data from different files to memory]
    2) upload all blocks to peer , one by one.

    getpiece(torrent, piece_index)
    -> Allocate memory for piece
    -> jump to (piece_index-1)*(piece structure size)
    -> loop over i ( i : 1 to num_files )
    -> open file for read
    -> seek file to file_offset
    -> skip piece_offset bytes in memory allocated for piece
    -> mov bytes from file to piece for length
    -> close file

    For future reference :
    [i]Generally, today's bittorrent client provides facility of downloading a single file (or set of selected files) out of complete torrent.

    This problem drills down to download set of pieces that belongs to that file.
    How to know which set of pieces to download for particular set of file(s) ?
    Possible Solution :
    Assign each file an index (starting from 1 to num_of_files) and store it within piece structure.
    Now, We can apply binary search over piece structure array and look which file index it contains.
    If file index is lesser or equal, search in left.
    Otherwise, search in right.
    Continue till you find piece that contains that file-index with corresponding file-offset 0.

    Credits :
    These are fruits of discussing with Dunkaist.


    @pathoswithin..
    I mean writing to allocated file in multiples of 16Mb. Hard drive spins and has big delay at operation start.
    Is 16Mb -> 16 Megabits -> 2000 kilo Bytes ?
    If it is the case, I doubt that I would be able to do that, as I will be writing one piece [512kB] to hard drive [may be to different files], due to strict memory requirements.

    @dunkaist...
       According to above strategy ,
    I ,now, need to point pieces to an array of piece structure , instead of , an array of 20-byte hash of pieces , right ?

    Code: Select all

    struct torrent
            trackers_cnt    dd ?
            trackers        dd ?    ; array
            peers_cnt       dd ?
            peers           dd ?
            pieces_cnt      dd ?
            pieces          dd ?
            piece_length    dd ?
            files_cnt       dd ?
            files           dd ?
            info            dd ?
            info_hash       rb 20   ; sha1 length
            peer_id         rb 20
            port            dd ?
            name            rb 256  ; utf-8
            uploaded        dd ?
            downloaded      dd ?
            left            dd ?
            pid             dd ?
            parent_pid      dd ?
            stack           dd ?
            stub            dd ?
            ipc_buf         dd ?
            ipc_msg         ipc_message
            proc_info       process_information
    ends
    
    And I need to pre-allocate memory of , piece_cnt*100 bytes, for "pieces" , right ?
    Thanks,
    Utsav Chokshi
  • I will be writing one piece [512kB] to hard drive [may be to different files], due to strict memory requirements
    Which requirements?

    Mentors
    I see, you have ignored my plea to communicate with student publicly.
  • Pathoswithin wrote:
    I will be writing one piece [512kB] to hard drive [may be to different files], due to strict memory requirements
    Which requirements?
    IMHO, this is something that should be configurable, as in any decent client.
    "Any intelligent fool can make things bigger, more complex, and more violent. It takes a touch of genius -- and a lot of courage -- to move in the opposite direction." Albert Einstein
  • Who is online

    Users browsing this forum: No registered users and 9 guests