See update below for important information on how to obtain a torrent file for any file on the Internet.

Update; get torrent for any file.

One disadvantage of all the tools mentioned below, except rsync, (which has its own difficulty; with rsync the server must be set up to allow anonymous connections to the rsync daemon.) is that they require you to have a valid copy of the original before you can prepare the 'helper' file that they rely upon. Hence, if the ordinator of the file does not supply this you are stuck. Or at least you were stuck.

Not any more. There is now an on-line service at which you can generate your own torrent file with which you can both verify an existing download or download it de nova.

If a torrent file does not exist create a torrent file at burnbit.com/

Now you can use this torrent file, in conjunction with aria2c, to check and correct a failed download or to download it from the start. e.g. : --

aria2c -l Windows-7-64bit-fromtorrent.log --check-integrity=true  -c http://msft.digitalrivercontent.net/win/X17-58997.iso \
    --torrent-file=X17-58997.iso.torrent

Introduction

I have been trying for many years to improve the way in which Linux Distros are being distributed on the Internet, mainly with the motive of saving time/data usage for the user, along with increasing the reliability of the process and reducing the need for large bandwidths on the servers and the space needed to store the files.

Tools

Among the contenders for honours in this regard are, in alphabetical order : --

aria2c

bittorrent  or tramsmission

jigdo

rsync

zsync

metalinks

mirrorbrain

Each of these tools has its own special usefulness and field of use : --

• aria2c is a versatile, threaded, downloading tool that can make use of metalinks, http, ftp and torrents to download large files. It does not reused existing data but does verify the download.
  
  
• jigdo is the official tool for distributing Debian ISO files. It works nicely with the main CD and DVD ISO files but not with the LiveCD or LiveDVD files, which have their main content in one squashfs file.
  
  
• rsync, when it is available at the server end, can sync any number of files and folders with just the changes being transmitted after the initial download.
  
  
• zsync is a tool specifically designed for updating large files, including ISO files, reusing existing data and downloading only changed or missing files or chunks of a file. It has the ability to look inside the ISO file and detect the actual files that reside within that structure. It does not require the re-downloading of the entire squashfs included in a LiveCD or LiveDVD but it does require that the entire target ISO file be hosted along with the .zsync file needed to search existing files for reusable data and to create and verify the target ISO file when all is complete.

Information about zsync : --

artistx@artistx:/media/Elements/LinuxDownloads/InstallationFiles/zsync$ ls -l
total 240
-rwxrwxrwx 1 artistx artistx 245592 2010-09-21 00:36 zsync-0.6.2.tar.bz2

Zsync update released

Monday, 20 September 2010

zsync 0.6.2 released

zsync 0.6.2 is now available from the download page.

This fixes a few bugs that have been spotted in the previous version:

• fix for using zsync client on files >2GB on 32bit systems.
• fix redirect handling.
• improve some edge cases dealing with unusual seed data patterns.
• optimise by stopping reading seed files if target file is complete.
• fix infinite loop in zsyncmake when given a truncated (invalid) .gz
• fix --disable-profile to configure.

Posted by Colin Phipps at 05:56 0 comments

Some other tools that aid in distributing large files.

• metalinks are text files that contain meta-information about a download, including the names of file(s), their checksums, their locations and various mirrors