Overview of using three tools for getting large files reliably in the first instance
- Download any file using aria2c with the following commands : --
if desired, create a new directory for the file(s) mkdir ArtistX-1.5 then locate in the new directory cd ArtistX-1.5
In the download directory issue the command aria2c -c http://bo.mirror.garr.it/mirrors/artistx/artistx_1.5_live_dvd_iso_15_09_2013.iso
If a .torrent file exists for this single file, use aria2c -c -T artistx-1.5-live-dvd.iso.15-09-2013.iso.Burnbit.torrent \
http://bo.mirror.garr.it/mirrors/artistx/artistx_1.5_live_dvd_iso_15_09_2013.iso
using both the torrent file and the http server as source of the target file. The .torrent file named above I downloaded from http://burnbit.com/torrent/291292/artistx_1_5_live_dvd_iso_15_09_2013_iso and renamed Using
mv "artistx 1 5 live dvd iso 15 09 2013 iso [Burnbit].torrent" artistx_1.5_live_dvd_iso_15_09_2013.iso.torrent
Downloading using a .torrent file from Burnbit will use both any swarm that exists and the http/ftp site that offers the original target file.
2. Download any file for which there is a .torrent file available, at the source of the target file, using 'transmission' or your favourite torrent agent.
3. Download any file for which there is a .zsync file available at the source of the target file using zsync, especially when you already have a nearly-similar file that you can update from, avoiding having to download at least some of the data in the target file.
For more, in-depth discussion of this, see below.
I spent hours working on this page but lost it all when moving away from the page despite having saved it continuously during the editing. This is a re-write from memory of what I had written.
How to recover a large download without downloading it all over again.
Many times when I have downloaded a large file, mostly ISO files containing a disk image of a Linux distribution, I have got to the end of the process only to discover that some small corruption had crept into the file. Having been in this situation more than a few times I have developed several methods for recovering without needing to download the entire file again. Sometimes that might be the easiest option, but then again there is no guarantee that the next download will not also fail. And sometimes the time taken might be more important than just saving bandwidth which has these days become almost too cheap to worry about saving.
Today I downloaded the ISO file of the latest ArtistX OS from http://artistx.org/blog/download/ and checking it found that it failed the validation test.
Verifying the download
Mostly, when offering a large file, the download site offers a checksum, md5sum, sha1sum, etc, to validate the download against. This is easily done on the command line in Linux with 'md5sum name-of-file.iso' or the like.
paul@paul-P5Q-SE /media/data/MyDownloadedISOFiles/ArtistX/artistx_1.5/artistx-1.5 $ date; time md5sum artistx_1.5_live_dvd_iso_15_09_2013.iso; date
Mon May 26 23:36:23 NZST 2014
591ccba3bc244e8137b3a5a1dcfb8ede artistx_1.5_live_dvd_iso_15_09_2013.iso
real 3m2.739s
user 0m8.405s
sys 0m2.528s
Mon May 26 23:39:26 NZST 2014
paul@paul-P5Q-SE /media/data/MyDownloadedISOFiles/ArtistX/artistx_1.5/artistx-1.5 $ cat artistx_1.5_live_dvd_iso_15_09_2013.iso.md5
f26a75da1f1eeca7f23356481dc825c8 artistx_1.5_live_dvd_iso_15_09_2013.iso
paul@paul-P5Q-SE /media/data/MyDownloadedISOFiles/ArtistX/artistx_1.5/artistx-1.5 $
The md5sums do not agree so the file has become corrupted during the download.
Tools for recovery
The following tools can be used to recover a corrupted file without the need to download it from the start. However they all need an additional file which contains information on chuncks of the target file together with checksums that can be individually checked and if the chunk does not validate it can be re-downloaded to overwrite the corrupted portion. When all the chunks have thus been validated the entire file will be validated and this will ensure a completely valid file with the minimum of data re-downloaded.
Aria2c and Transmission (or your favourite torrent agent) both require the same .torrent file which can be supplied by the original site offering the target file, or when it is missing you can create your own .torrent using a free, on-line service at http://burnbit.com . I created my own .torrent file there because the torrent file offered at the ArtistX download page is a multi-file torrent which is not suitable for a single-file validation. The URL for my .torrent file is : --
http://burnbit.com/download/291292/artistx_1_5_live_dvd_iso_15_09_2013_iso
aria2c
aria2c is my most-favoured download tool for large files. It is very reliable and fast, very seldom giving a corrupted file at the end of the process, and if used in conjunction with a .torrent file will even fix any errors that may have crept in when using another tool.
Note : In this exemplar I have created a separate directory in which to work and copied files into it. That makes the process fail-safe, especially if you are new to the command line where mistakes are frequent and can be catastrophic!
Background Information on aria2c
For information on using aria2c use the man file, 'man aria2c' for a very full technical description of all the capabilities of this software.
SEE ALSO
Project Web Site: http://aria2.sourceforge.net/
aria2 Wiki: http://sourceforge.net/apps/trac/aria2/wiki
Metalink Homepage: http://www.metalinker.org/
The Metalink Download Description Format: http://tools.ietf.org/html/rfc5854
Here are the commands (in bold) and terminal output following them : --
paul@paul-P5Q-SE /media/data/MyDownloadedISOFiles/ArtistX $ cd artistx-1.5/
paul@paul-P5Q-SE /media/data/MyDownloadedISOFiles/ArtistX/artistx-1.5 $ ls -l
total 3942352
-rw-r--r-- 1 paul paul 4025106432 May 27 12:57 artistx_1.5_live_dvd_iso_15_09_2013.iso
-rw-r--r-- 1 paul paul 38886 May 27 11:39 artistx-1.5-live-dvd.iso.15-09-2013.iso.Burnbit.torrent
-rw-r--r-- 1 paul paul 74 May 26 20:51 artistx_1.5_live_dvd_iso_15_09_2013.iso.md5
-rw-r--r-- 1 paul paul 7861832 May 27 11:58 artistx_1.5_live_dvd_iso_15_09_2013.iso.zsync
drwxr-xr-x 2 paul paul 4096 May 27 12:50 Testing-zsync
paul@paul-P5Q-SE /media/data/MyDownloadedISOFiles/ArtistX/artistx-1.5 $ mkdir Testing-aria2c
paul@paul-P5Q-SE /media/data/MyDownloadedISOFiles/ArtistX/artistx-1.5 $ cp artistx_1.5_live_dvd_iso_15_09_2013.iso Testing-aria2c/
paul@paul-P5Q-SE /media/data/MyDownloadedISOFiles/ArtistX/artistx-1.5 $ cp artistx-1.5-live-dvd.iso.15-09-2013.iso.Burnbit.torrent Testing-aria2c/
paul@paul-P5Q-SE /media/data/MyDownloadedISOFiles/ArtistX/artistx-1.5 $ cp artistx_1.5_live_dvd_iso_15_09_2013.iso.md5 Testing-aria2c/
paul@paul-P5Q-SE /media/data/MyDownloadedISOFiles/ArtistX/artistx-1.5 $ cd Testing-aria2c/
paul@paul-P5Q-SE /media/data/MyDownloadedISOFiles/ArtistX/artistx-1.5/Testing-aria2c $ ls -l
total 3934656
-rw-r--r-- 1 paul paul 4025106432 May 27 16:08 artistx_1.5_live_dvd_iso_15_09_2013.iso
-rw-r--r-- 1 paul paul 38886 May 27 16:15 artistx-1.5-live-dvd.iso.15-09-2013.iso.Burnbit.torrent
-rw-r--r-- 1 paul paul 74 May 27 16:15 artistx_1.5_live_dvd_iso_15_09_2013.iso.md5
paul@paul-P5Q-SE /media/data/MyDownloadedISOFiles/ArtistX/artistx-1.5/Testing-aria2c $ aria2c --check-integrity artistx-1.5-live-dvd.iso.15-09-2013.iso.Burnbit.torrent
2014-05-27 16:33:21.184005 NOTICE - IPv4 DHT: listening to port 6887
*** Download Progress Summary as of Tue May 27 16:34:21 2014 ***
========================================================================================================================================================================================
[#1 SIZE:0B/3,838.6MiB(0%) CN:0 SEED:0 SPD:0Bs]
FILE: /media/data/MyDownloadedISOFiles/ArtistX/artistx-1.5/Testing-aria2c/artistx_1.5_live_dvd_iso_15_09_2013.iso
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[#1 SIZE:0B/3,838.6MiB(0%) CN:0 SEED:0 SPD:0Bs] [Checksum:#1 3,820.0MiB/3,838.6MiB(99%)]
2014-05-27 16:35:18.530004 NOTICE - Verification finished successfully. file=/media/data/MyDownloadedISOFiles/ArtistX/artistx-1.5/Testing-aria2c/artistx_1.5_live_dvd_iso_15_09_2013.iso
2014-05-27 16:35:18.585278 NOTICE - IPv4 BitTorrent: listening to port 6928
2014-05-27 16:35:18.591475 ERROR - IPv6 BitTorrent: failed to bind port 6928
Exception: [SocketCore.cc:308] errorCode=1 Failed to bind a socket, cause: Name or service not known
*** Download Progress Summary as of Tue May 27 16:35:21 2014 ***
========================================================================================================================================================================================
[#1 SEEDING(ratio:0.0) CN:0 SEED:0]
(truncated)
*** Download Progress Summary as of Tue May 27 16:39:24 2014 ***
========================================================================================================================================================================================
[#1 SEEDING(ratio:0.0) CN:0 SEED:0]
FILE: /media/data/MyDownloadedISOFiles/ArtistX/artistx-1.5/Testing-aria2c/artistx_1.5_live_dvd_iso_15_09_2013.iso
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[#1 SEEDING(ratio:0.0) CN:0 SEED:0]
2014-05-27 16:39:33.009174 NOTICE - Shutdown sequence commencing... Press Ctrl-C again for emergency shutdown.
2014-05-27 16:39:33.024183 NOTICE - Download complete: /media/data/MyDownloadedISOFiles/ArtistX/artistx-1.5/Testing-aria2c/artistx_1.5_live_dvd_iso_15_09_2013.iso
2014-05-27 16:39:33.024225 NOTICE - Your share ratio was 0.0, uploaded/downloaded=0B/3,838.6MiB
Download Results:
gid|stat|avg speed |path/URI
===+====+===========+===========================================================
1| OK| 0B/s|/media/data/MyDownloadedISOFiles/ArtistX/artistx-1.5/Testing-aria2c/artistx_1.5_live_dvd_iso_15_09_2013.iso
Status Legend:
(OK):download completed.
paul@paul-P5Q-SE /media/data/MyDownloadedISOFiles/ArtistX/artistx-1.5/Testing-aria2c $
transmission
Transmission is a good tool for downloading Linux distribution ISO files which often have a .torrent file available at the original download site. That would be the first choice if it is available. As mentioned, if there is no .torrent file provided you can create your own one at the Burnbit site. However, if you create your own torrent file it will not work well with transmission due to lack of a swarm of other users to assist the download. In that case it is better to use aria2c with the torrent file as it will then download both from the original http or ftp site and from any torrent swarm.
In any case, if you have downloaded an ISO file and find, on checking its checksum, that it fails the check, then transmission can be used to recover the download without starting from scratch again.
If the file was first downloaded with another tool and the .torrent file contains more than one file, creating its own sub-folder in which to put the files, then just start transmission to create the folder, pause it, then copy the failed file into the folder and get transmission to verify the local data. You do this from the Menu Bar with : --
After selecting the torrent in transmission, use > Torrent > Verify Local Data
Transmission will check through the file, chunk by chunk, and only download any chunks that fail the checksum validation test.
zsync
zsync has its own file format and constructs a .zsync file from the target file using 'zsyncmake' that will be installed if zsync is installed. The one drawback of using zsync, as I have found over the years, is that few people bother to create and offer a zsync file along with the original file to download. This is the greatest limitation of the zsync method and one that cannot be got around. Hence this method has been relegated to last place in my preferences. It still can be useful in some circumstances. It is possible for someone who has a validated copy of the file that you want to create quite easily a .zsync file and send it to you as an attachment to an email. This attached file will be in the order of 0.2% of the target file. For ArtistX_1.5 which is 3.8 GB in size, the zsync file is only 7.5M - still a significant size but probably manageable as an attachment with Gmail. That or ftp it to an on-line site and send the link.
Here are the commands and terminal output following by which I verified my local file against a valid copy using a .zsync file : --
paul@paul-P5Q-SE /media/data/MyDownloadedISOFiles/ArtistX/artistx-1.5/Testing-zsync $ ls -l
total 3942304
-rw------- 1 paul paul 4025106432 May 27 12:48 artistx_1.5_live_dvd_iso_15_09_2013.iso.part
-rw-r--r-- 1 paul paul 7861832 May 27 12:39 artistx_1.5_live_dvd_iso_15_09_2013.iso.zsync
paul@paul-P5Q-SE /media/data/MyDownloadedISOFiles/ArtistX/artistx-1.5/Testing-zsync $ zsync artistx_1.5_live_dvd_iso_15_09_2013.iso.zsync
reading seed file artistx_1.5_live_dvd_iso_15_09_2013.iso: *****************************************************************************************(truncated)
Read artistx_1.5_live_dvd_iso_15_09_2013.iso. Target 100.0% complete. ****************************************
verifying download...checksum matches OK
used 4025106432 local, fetched 0
paul@paul-P5Q-SE /media/data/MyDownloadedISOFiles/ArtistX/artistx-1.5/Testing-zsync $ ls -l
total 7876916
-rw------- 1 paul paul 4025106432 May 26 22:54 artistx_1.5_live_dvd_iso_15_09_2013.iso
-rw------- 1 paul paul 4025106432 May 26 22:54 artistx_1.5_live_dvd_iso_15_09_2013.iso.zs-old
-rw-r--r-- 1 paul paul 7861832 May 27 12:39 artistx_1.5_live_dvd_iso_15_09_2013.iso.zsync
paul@paul-P5Q-SE /media/data/MyDownloadedISOFiles/ArtistX/artistx-1.5/Testing-zsync $
You will see that zsync kindly creates a new file and renames the older file with the suffix '.zs-old' so that it has its own fail-safety feature built in. Since I was testing on a known good file the target is shown as '100.0% complete' and 'used 4025106432 local, fetched 0'. Also the overall checksum of the total file was checked and found to be 'OK'.
Comments (0)
You don't have permission to comment on this page.