Board.KolibriOS.org

Официальный форум KolibriOS
Текущее время: Сб июл 22, 2017 7:48 am

Часовой пояс: UTC+03:00




Начать новую тему  Ответить на тему  [ 23 сообщения ]  На страницу 1 2 След.
Автор Сообщение
 Заголовок сообщения: GSoC : BitTorrent
СообщениеДобавлено: Вт май 31, 2016 11:43 am 
Не в сети
KSoC/GSoC Student

Зарегистрирован: Чт мар 03, 2016 3:50 pm
Сообщения: 21
Hello All,

I will be posting all my updates and weekly status as reply to this post.

I am attaching my GSoC proposal.

Following is my Github repository :
https://github.com/ChokshiUtsav/BitTorrent

Currently following milestones have been reached :
1 : Decoding bencoded torrent file and extracting meta info : Partially Tested
2 : Generating HTTP req and Parsing HTTP response : Partially Tested
3 : Establishing connection with one of peers : Partially implemented.

Thanks,
Utsav Chokshi
IRC Name : Utsav_Chokshi
Developer Chat ID : Utsav_Chokshi


Вложения:
Utsav_Chokshi_GSoCProjectProposal_Final.pdf [153.66 КБ]
278 скачиваний
Вернуться к началу
 Заголовок сообщения: Re: GSoC : BitTorrent
СообщениеДобавлено: Вт июн 07, 2016 3:06 pm 
Не в сети
KSoC/GSoC Student

Зарегистрирован: Чт мар 03, 2016 3:50 pm
Сообщения: 21
Hello All,

This is my weekly report :
https://docs.google.com/document/d/1bMl8OpDI7E1mcFNZs0duIINAXrRCX7HrqghOeBlBzrQ/edit?usp=sharing

And I have some queries and I will post it here soon.


Вернуться к началу
 Заголовок сообщения: Re: GSoC : BitTorrent
СообщениеДобавлено: Сб июн 11, 2016 12:55 pm 
Не в сети
Mentor/Kernel Developer
Аватара пользователя

Зарегистрирован: Пт июн 30, 2006 9:01 am
Сообщения: 1211
Utsav_Chokshi писал(а):
I am not able to connect with peers. [check peer.asm]
I have verified my code by connecting to well-known IPs (telehack.com) and it works. Somehow I feel that there is problem with my net configuration.

Dunkaist sent me a patch for his torrent code the other day, related to the fact that port numbers for socket functions must be in network byte order, like on most other platforms. Is/was your problem perhaps related to this?

Utsav_Chokshi писал(а):
I don’t know how to make a daemon/service in kolibrios which makes code to run in background. Any pointers ?

Easy: Dont draw a GUI, if you want to hide from taskbar the first name character of the program name should be the @ sign, use IPC or local sockets to communicate with the program(s).

Utsav_Chokshi писал(а):
Good examples for building UI.

This one is trickier, the controls you have sketched do not really exist yet for kolibriOS.

_________________
"Any intelligent fool can make things bigger, more complex, and more violent. It takes a touch of genius -- and a lot of courage -- to move in the opposite direction." Albert Einstein


Вернуться к началу
 Заголовок сообщения: Re: GSoC : BitTorrent
СообщениеДобавлено: Сб июн 11, 2016 6:37 pm 
Не в сети
Mentor/Kernel Developer
Аватара пользователя

Зарегистрирован: Чт мар 26, 2015 5:16 pm
Сообщения: 1163
Utsav_Chokshi
Цитата:
the controls you have sketched do not really exist yet for kolibriOS
Unfortunately, the Widget toolkit task was not chosen this GSoC, but if you will write some GUI element as an external module, we will appreciate that. You also may try to cooperate with nisargshah95 about this.


Вернуться к началу
 Заголовок сообщения: Re: GSoC : BitTorrent
СообщениеДобавлено: Сб июн 11, 2016 9:03 pm 
Не в сети
KSoC/GSoC Student

Зарегистрирован: Чт мар 03, 2016 3:50 pm
Сообщения: 21
@Pathoswithin ..

Цитата:
Unfortunately, the Widget toolkit task was not chosen this GSoC, but if you will write some GUI element as an external module, we will appreciate that. You also may try to cooperate with nisargshah95 about this.


My GUI includes few buttons, open dialog, scroll bars and edit boxes. And I am using boxlib and proclib for that. I found it sufficient. Only control, I have not tried to incorporate yet is ScrollBar.
And yes I am in touch with NisargShah95.

@hidnplayr..
Thanks for answers.
Цитата:
Easy: Dont draw a GUI, if you want to hide from taskbar the first name of the program should be the @ sign, use IPC or local sockets to communicate with the program(s).


I got the idea of not drawing GUI for program to run in background.
But I did not understand sentence : "the first name of the program should be the @ sign" ?

As per discussuion with dunkaist,
For communication between torrent client UI and background program/daemon, using network socket seems preferable. As UI and daemon can be run on two different kolibrios boxes. Your suggestion and inputs are welcome.
Цитата:
Dunkaist sent me a patch for his torrent code the other day, related to the fact that port numbers for socket functions must be in network byte order, like on most other platforms. Is/was your problem perhaps related to this?


Yes I have received patch from dunkaist and I am now successfully able to communicate[handshake] with peers.
Yes, It was problem related byte order.

I have made some other progress as well.
I'll update this thread along with some design quesions.

Thanks.


Вернуться к началу
 Заголовок сообщения: Re: GSoC : BitTorrent
СообщениеДобавлено: Сб июн 11, 2016 9:15 pm 
Не в сети

Зарегистрирован: Вт мар 08, 2016 11:00 pm
Сообщения: 248
Utsav_Chokshi писал(а):
As per discussuion with dunkaist,
For communication between torrent client UI and background program/daemon, using network socket seems preferable. As UI and daemon can be run on two different kolibrios boxes. Your suggestion and inputs are welcome.

See my attached example using shared mem functions.
Its c-language, but using asm is easy too. SysFn 68.22


Вложения:
shm_run.c [735 байт]
14 скачиваний
runner.c [1.01 КБ]
12 скачиваний
Вернуться к началу
 Заголовок сообщения: Re: GSoC : BitTorrent
СообщениеДобавлено: Сб июн 11, 2016 9:25 pm 
Не в сети
Mentor/Kernel Developer
Аватара пользователя

Зарегистрирован: Чт мар 26, 2015 5:16 pm
Сообщения: 1163
*the first char of the name should be @


Вернуться к началу
 Заголовок сообщения: Re: GSoC : BitTorrent
СообщениеДобавлено: Ср июн 15, 2016 9:43 am 
Не в сети
KSoC/GSoC Student

Зарегистрирован: Чт мар 03, 2016 3:50 pm
Сообщения: 21
Hello,

Following is over-all torrent client design I can think of :
[img]
Вложение:
bittorrent_design.jpg
bittorrent_design.jpg [ 238.53 КБ | 2358 просмотров ]

[/img]

Explanation :
FrontEnd.asm : GUI of application and consumes one thread
BackEnd.asm : Daemon/Backend of torrent application and consumes two thread

FronEnd and BackEnd communicates through network sockets over TCP protocol via server[backend]-client[frontend] mechanism.
TCP Protocol payload format [from frontend -> backend] is :
<LengthOfMessage><TypeOfMessage><PayloadOfMessage> [4Bytes-1Byte-Nbytes]
Type of Message can be :
TORRENT_ADD(1) (Payload : Torrent_ID : Torrent File Path : Download Location)
TORRENT_START(2) (Payload : Torrent_ID)
TORRENT_PAUSE(3) (Payload : Torrent_ID)
TORRENT_REMOVE(4) (Payload : Torrent_ID)
TORRENT_SHOW(5) (Payload : Torrent_ID)
TORRENT_QUIT(6) (This message closes backend)

TCP Protocol Payload format for messages[ from backend -> frontend ] is :
<LengthOfMessage><PayloadOfMessage>

In frontend:
There is only one thread which draws UI as well as connects with backend and updates it.

In backend :
Thread one handles all incoming request from frontend and provides neccessary information.
Thread two handles all torrents' downlaoding and uploading.

These two threads of backend share one common data-area [/list] : List of Torrents.
Thread one mainly consumes this list and add/remove torrents
Thread two mainly updates this list.
Each element of list follows torrent structure described in torrent.inc [Few entries like torrent_id , is yet to be made]

-> Thread two loops over all torrents in torrent list.
-> For each torrent, it works for every torrernt for around 5000ms. [this time is subject to experiment and yet to optimize].
-> For each torrent, it connects with tracker server [if timeout is happended for connecting with tracker server , which is usually 1800 secs (30 mins)]
it connects with all active peers, offering pieces to download.
it sends keep-alive messages to inactive peers [2 minute is max ideal period] or close connection with them.
it accepts incoming connections from other peers [at max 16 peers needs to be handled , according to torrent guideline ] and uploads pieces to them.
-> Uploading and downloading starts at two timeout events within 5000 ms span.

Credits :
-> Idea of using network socket over IPC mechanism is conclusion of discussion with Dunkaist.
Benefit of using network socket over IPC , is , that we can have backend and frontend on two different kolibrios boxes.

-> Keeping thread numbers as minimal as possible is conclusion of discussion with Hidnplayr.
As kolibrios , as a whole , supports only 256 threads.

-> Keeping timeouts for handling multiple connections is also concluson of discussion with Hidnplayr.
So that fairness can be ensured.

Help required :
-> Currently I am not able to connect backend with frontend using localhost as IP [127.0.0.1]. My program gets stuck at connect system call with interrupt 40h [provided server is up]. Any help ?
-> I have attached sample tcp-server and tcp-client programs for the same.

Вложение:
tcp_client.asm [3.45 КБ]
18 скачиваний

Вложение:
tcp_server.asm [4.65 КБ]
14 скачиваний


-> Currently, I am able to open file browser dialog and copies a path of selected file to editbox.
-> Problem, I am facing is : I am not able to select folder. [I need this for download location of torrent]. I guess it has something to do with filter of open dialog.

Current Progress :
-> I am able to download one piece-block from peer.
-> Basic structure of GUI is almost ready and I am much clear regarding the over-all design.

Github Link : https://github.com/chokshiutsav/bittorrent

Suggestion regarding any desing aspect is welcome. :)

Thanks,
Utsav Chokshi


Вернуться к началу
 Заголовок сообщения: Re: GSoC : BitTorrent
СообщениеДобавлено: Пн июн 20, 2016 2:11 pm 
Не в сети
KSoC/GSoC Student

Зарегистрирован: Чт мар 03, 2016 3:50 pm
Сообщения: 21
Subject : Regarding writing pieces of file being downloaded and merging them in a single file

As per discussion with ashmew2 @IRC,

- A piece of torrent file can be as large as 512-1024kB , which in turn gets downloaded as blocks of 4kBs.
- So strategy I decided is , downloading blocks of single piece from single peer in a sequence .
- Keep Writing all those blocks in separate piece-file.
- Once, the complete piece is downloaded. It is verified against its Hash. [P.S. Hash of each piece is stored in ".torrent" file]
- We need to write that piece to bigger file [Or you can say merge with other pieces].
- For that, a file (call this "File X") equal to original file-size needs to be created before download starts.
- As length of each piece is same and known before-hand and index of piece is also known , we can write piece to File X at specifc
offset [(index-1)*(piece-length)].

Here I am attaching demo program which replicates last two steps.
Вложение:
fileio_bigsize.asm [3.28 КБ]
13 скачиваний


Explanation for demo program :
It creates file of size 100kb using Function 70 , subfunction 2 and 4.
It writes 8 byte of data at offset 128 and offset 256 with respect to start position (i.e. byte 0)

@ashmew2...Function 70, subfunction 4 writes 0 to all extended bytes by default. :) [With reference to our discussion of initialzing file with zero]

~Utsav Chokshi


Вернуться к началу
 Заголовок сообщения: Re: GSoC : BitTorrent
СообщениеДобавлено: Пн июн 20, 2016 7:39 pm 
Не в сети
Mentor/Kernel Developer
Аватара пользователя

Зарегистрирован: Чт мар 26, 2015 5:16 pm
Сообщения: 1163
Note, that for big files optimal data block size for sysfunction 70.3 is 16 Mb.

Also note, that currently, for performance reasons, sysfunction 70.4 on NTFS erases up to 16 Mb, and just extends if operation is bigger (and not tested yet).


Вернуться к началу
 Заголовок сообщения: Re: GSoC : BitTorrent
СообщениеДобавлено: Вт июн 21, 2016 4:50 pm 
Не в сети
KSoC/GSoC Student

Зарегистрирован: Чт мар 03, 2016 3:50 pm
Сообщения: 21
Pathoswithin писал(а):
Note, that for big files optimal data block size for sysfunction 70.3 is 16 Mb.


What do you exactly mean by data block size ?
Do you mean, allocating file-size in multiples of 16Mb ?


Вернуться к началу
 Заголовок сообщения: Re: GSoC : BitTorrent
СообщениеДобавлено: Вт июн 21, 2016 6:08 pm 
Не в сети
Mentor/Kernel Developer
Аватара пользователя

Зарегистрирован: Чт мар 26, 2015 5:16 pm
Сообщения: 1163
I mean writing to allocated file in multiples of 16Mb. Hard drive spins and has big delay at operation start.


Вернуться к началу
 Заголовок сообщения: Re: GSoC : BitTorrent
СообщениеДобавлено: Пт июн 24, 2016 4:40 pm 
Не в сети
KSoC/GSoC Student

Зарегистрирован: Чт мар 03, 2016 3:50 pm
Сообщения: 21
Hello All,

Subject : Single Piece Download and Upload Strategy and Overview of file operations

Before Bittorrent Client starts contacting peers :

1) Allocate file-space :
[In case of multi-file torrent]
1) Create directory. Name of directory is provided in .torrent file and stored in torrent structure.
2) Create empty files of size equal to files to be downloaded.
[In case of single-file torrent]
1) Create empty file of size equal to file to be downloaded.

1) Fill following piece structure :
An array of piece structure is created and filled.
Код:
struct piece
        index                      dd ?
        piece_hash                rb 20
        num_files               dd ?  ;array_size and it defines number of files' data contained by this piece
        piece_offset            dd ?  ;array
        length                     dd ?  ;array
        file_offset                 dd ?  ;array
        file_names                 dd ?  ;array
        num_blocks_downloaded   dd ?
        download_status         dd ?
ends

BT_PIECE_DOWNLOAD_NOT_STARTED = 0
BT_PIECE_DOWNLOAD_IN_PROGRESS = 1
BT_PIECE_DOWNLOAD_COMPLETE    = 2


Please note that,
-> Few member variables , like index and download_status , may be redundant.
As piece structure is stored in form of an array , so index may not be needed to store.
As bitfiled of overall torrent is stored, download_status may not be needed to store.

-> Array of piece structure will be kept in memory.
How much is memory required ? => For 2GB of torrent file roughly 300-500kB is required.

Download a piece :
1) Download all blocks of piece from single peer.
2) Kepp all blocks of piece in memory till complete piece is downloaded [It requires 512kB (standard piece size) memory]
3) Create SHA1 hash of piece [createSHA1Hash()]
4) getHashOfPiece()
5) VerifyHashOfPiece()
6) setpiece(torrent,piece_index) [it wirtes piece to appropriate files]

setpiece(torrent,piece_index)
-> jump to (piece_index-1)*(piece structure size)
-> loop over i ( i : 1 to num_files )
-> open file for write
-> seek file[i] to file_offset[i]
-> skeip piece_offset[i] bytes in piece
-> mov bytes from piece to file for length[i]
-> close file
-> delete piece

Upload a piece :
1) getpiece(torrent, piece_index) [it brings piece data from different files to memory]
2) upload all blocks to peer , one by one.

getpiece(torrent, piece_index)
-> Allocate memory for piece
-> jump to (piece_index-1)*(piece structure size)
-> loop over i ( i : 1 to num_files )
-> open file[i] for read
-> seek file[i] to file_offset[i]
-> skip piece_offset[i] bytes in memory allocated for piece
-> mov bytes from file to piece for length[i]
-> close file

For future reference :
[i]Generally, today's bittorrent client provides facility of downloading a single file (or set of selected files) out of complete torrent.

This problem drills down to download set of pieces that belongs to that file.
How to know which set of pieces to download for particular set of file(s) ?
Possible Solution :
Assign each file an index (starting from 1 to num_of_files) and store it within piece structure.
Now, We can apply binary search over piece structure array and look which file index it contains.
If file index is lesser or equal, search in left.
Otherwise, search in right.
Continue till you find piece that contains that file-index with corresponding file-offset 0.

Credits :
These are fruits of discussing with Dunkaist.



@pathoswithin..
Цитата:
I mean writing to allocated file in multiples of 16Mb. Hard drive spins and has big delay at operation start.

Is 16Mb -> 16 Megabits -> 2000 kilo Bytes ?
If it is the case, I doubt that I would be able to do that, as I will be writing one piece [512kB] to hard drive [may be to different files], due to strict memory requirements.



@dunkaist...
[area=]
According to above strategy ,
I ,now, need to point pieces to an array of piece structure , instead of , an array of 20-byte hash of pieces , right ?
Код:
struct torrent
        trackers_cnt    dd ?
        trackers        dd ?    ; array
        peers_cnt       dd ?
        peers           dd ?
        pieces_cnt      dd ?
        pieces          dd ?
        piece_length    dd ?
        files_cnt       dd ?
        files           dd ?
        info            dd ?
        info_hash       rb 20   ; sha1 length
        peer_id         rb 20
        port            dd ?
        name            rb 256  ; utf-8
        uploaded        dd ?
        downloaded      dd ?
        left            dd ?
        pid             dd ?
        parent_pid      dd ?
        stack           dd ?
        stub            dd ?
        ipc_buf         dd ?
        ipc_msg         ipc_message
        proc_info       process_information
ends


And I need to pre-allocate memory of , piece_cnt*100 bytes, for "pieces" , right ?
[/area]

Thanks,
Utsav Chokshi


Вернуться к началу
 Заголовок сообщения: Re: GSoC : BitTorrent
СообщениеДобавлено: Пт июн 24, 2016 9:17 pm 
Не в сети
Mentor/Kernel Developer
Аватара пользователя

Зарегистрирован: Чт мар 26, 2015 5:16 pm
Сообщения: 1163
Цитата:
I will be writing one piece [512kB] to hard drive [may be to different files], due to strict memory requirements
Which requirements?

Mentors
I see, you have ignored my plea to communicate with student publicly.


Вернуться к началу
 Заголовок сообщения: Re: GSoC : BitTorrent
СообщениеДобавлено: Вс июн 26, 2016 10:25 pm 
Не в сети
Mentor/Kernel Developer
Аватара пользователя

Зарегистрирован: Пт июн 30, 2006 9:01 am
Сообщения: 1211
Pathoswithin писал(а):
Цитата:
I will be writing one piece [512kB] to hard drive [may be to different files], due to strict memory requirements
Which requirements?


IMHO, this is something that should be configurable, as in any decent client.

_________________
"Any intelligent fool can make things bigger, more complex, and more violent. It takes a touch of genius -- and a lot of courage -- to move in the opposite direction." Albert Einstein


Вернуться к началу
Показать сообщения за:  Поле сортировки  
Начать новую тему  Ответить на тему  [ 23 сообщения ]  На страницу 1 2 След.

Часовой пояс: UTC+03:00


Кто сейчас на конференции

Сейчас этот форум просматривают: нет зарегистрированных пользователей и 1 гость


Вы не можете начинать темы
Вы не можете отвечать на сообщения
Вы не можете редактировать свои сообщения
Вы не можете удалять свои сообщения
Вы не можете добавлять вложения

Найти:
Перейти:  
cron
Создано на основе phpBB® Forum Software © phpBB Limited
Русская поддержка phpBB