Monday, March 29 2010
I have just finished downloading Sabayon Linux 5.2 Gnome via a bit torrent and am running it as a LiveDVD now. It is a nice-looking distro and rather interesting from a number of perspectives.
One particular perspective is the method of supplying the download itself, which makes use of both mirrors and torrents as well as MetaLinks http://www.metalinker.org/why.html. While finding out about MetaLinks I also came across aria2 which is what I really want to talk about.
As you may be aware, I have pushed the use of zsync on a number of occasions and a number of forums. And although there has been some interest in this application the fact that it needs a bit of extra work for the publisher to publish the .zsync file along with the ISO file tends to make it an uphill battle to get the support this system needs at the supply end. Also, because it is a CLI tool it tends to be used by few at the user end also. Hence, despite my best efforts there has been a very low level of actual use of the zsync method of downloading.
That's why the aria2 project and the MetaLinks project have caught my attention. Aria2 is a downloading application that is aware of multiple protocols and can use them simultaneously. This can maximise the download speed. Not only that, but aria2, like zsync, can verify chunks of data as they are downloaded and can auto-correct any errors that may occur along the downloading process. Therefore it has advantages both for high-speed and low-speed Internet connections.
[url=http://www.debianadmin.com/aria2-high-speed-command-line-download-utility-in-debian.html]Aria2 - high speed command line download utility in Debian[/url]
[quote]aria2 is a utility for downloading files. The supported protocols are HTTP(S), FTP, BitTorrent (DHT, PEX, MSE/PE), and Metalink. It can download a file from multiple sources/protocols and tries to utilize your maximum download bandwidth.
It even supports downloading a file from HTTP(S)/FTP and BitTorrent at the same time, while the data downloaded from HTTP(S)/FTP is uploaded to the BitTorrent swarm. Using Metalink’s chunk checksums, aria2 automatically validates chunks of data while downloading a file like BitTorrent.[/quote]
The other thing in its favour is that, unlike zsync which has stagnated for years, this line of development seems to be much better supported and progressing.
More later ...
The Ultra Fast Download Utility
Installed in ArtistX 0.7
paul@paul-ArtistX07:~$ sudo apt-get install aria2
[sudo] password for paul:
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
libstlport4.6ldbl lp-solve virtualbox-ose-source
Use 'apt-get autoremove' to remove them.
The following extra packages will be installed:
libc-ares2
The following NEW packages will be installed:
aria2 libc-ares2
0 upgraded, 2 newly installed, 0 to remove and 2 not upgraded.
Need to get 1321kB of archives.
After this operation, 3736kB of additional disk space will be used.
Do you want to continue [Y/n]? y
Get:1 http://nz.archive.ubuntu.com jaunty/universe libc-ares2 1.5.2-4 [40.6kB]
Get:2 http://nz.archive.ubuntu.com jaunty/universe aria2 1.1.2-1 [1281kB]
Fetched 1321kB in 5s (241kB/s)
Selecting previously deselected package libc-ares2.
(Reading database ... 490833 files and directories currently installed.)
Unpacking libc-ares2 (from .../libc-ares2_1.5.2-4_i386.deb) ...
Selecting previously deselected package aria2.
Unpacking aria2 (from .../aria2_1.1.2-1_i386.deb) ...
Processing triggers for man-db ...
Setting up libc-ares2 (1.5.2-4) ...
Setting up aria2 (1.1.2-1) ...
Processing triggers for libc6 ...
ldconfig deferred processing now taking place
localepurge: Disk space freed in /usr/share/locale: 872K
paul@paul-ArtistX07:~$
Other aspects : --
Also looking at Metalinks and MirrorBrain.
Metalinks provides a small download file that orchestrates the download of a larger file or file-set.
MirrorBrain oversees the distribution of mirrors, redirects requests to optimise the client and load-share on the servers. It can generate Metalinks on request.
Test-run of aria2.
The command to run aria2 is 'aria2c'. Here is a trial run downloading a rather large ISO file. I tried using the torrent first of all, but the time to complete was enormous, so I stopped the process and re-started it with the URL for one of the mirrors. You can supply more than one URL but I didn't try that. Here is some of the output [Note aria2c is very verbose] : --
Showing the contents of the torrent file
paul@paul-ArtistX07:/media/Elements/LinuxDownloads/DownloadedISOfiles/Sabayon$ aria2c --dir=/media/Elements/LinuxDownloads/DownloadedISOfiles/Sabayon -S ./Sabayon_Linux_5.2_x86_G.iso.torrent
>>> Printing the contents of file './Sabayon_Linux_5.2_x86_G.iso.torrent'...
*** BitTorrent File Information ***
Mode: Single File Torrent
Announce:
http://tracker.sabayon.org/tracker.php/announce
Info Hash: bd632b8be37e8eff7238926f9b64cf091cd67c74
Piece Length: 1.0MiB
The Number of Pieces: 1681
Total Length: 1,680.1MiB
Files:
idx|path/length
===+===========================================================================
1|Sabayon_Linux_5.2_x86_G.iso
|1,680.1MiB
---+---------------------------------------------------------------------------
No files to download.
Starting the process to download the ISO using the torrent.
*****************************************************************************************************************************************************************************************************************************************************
paul@paul-ArtistX07:/media/Elements/LinuxDownloads/DownloadedISOfiles/Sabayon$ date && time aria2c --dir=/media/Elements/LinuxDownloads/DownloadedISOfiles/Sabayon Sabayon_Linux_5.1-r1_x86_GAMING.iso.torrent
Mon Mar 29 17:33:37 NZDT 2010
2010-03-29 17:33:37.224515 WARN - You may encounter the certificate verification error with HTTPS server. See --ca-certificate and --check-certificate option.
*** Download Progress Summary as of Mon Mar 29 17:34:37 2010 ***
==============================================================================================================================================================================================
[#1 SIZE:0B/3,733.5MiB(0%) CN:0 SPD:0.00KiB/s]
FILE: /media/Elements/LinuxDownloads/DownloadedISOfiles/Sabayon/Sabayon_Linux_5.1-r1_x86_GAMING.iso
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
*** Download Progress Summary as of Mon Mar 29 17:35:37 2010 ***
==============================================================================================================================================================================================
[#1 SIZE:0B/3,733.5MiB(0%) CN:0 SPD:0.00KiB/s]
FILE: /media/Elements/LinuxDownloads/DownloadedISOfiles/Sabayon/Sabayon_Linux_5.1-r1_x86_GAMING.iso
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
*** Download Progress Summary as of Mon Mar 29 17:36:38 2010 ***
==============================================================================================================================================================================================
[#1 SIZE:592.0KiB/3,733.5MiB(0%) CN:22 SPD:19.53KiB/s ETA:54h21m57s]
FILE: /media/Elements/LinuxDownloads/DownloadedISOfiles/Sabayon/Sabayon_Linux_5.1-r1_x86_GAMING.iso
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Note the long ETA! I stopped the process and re-started downloading directly from a mirror.
*****************************************************************************************************************************************************************************************************************************************************************************************************************************
Stopping the process
*** Download Progress Summary as of Mon Mar 29 18:59:39 2010 ***
==============================================================================================================================================================================================
[#1 SIZE:73.4MiB/3,733.5MiB(1%) CN:11 SPD:17.08KiB/s ETA:60h56m15s]
FILE: /media/Elements/LinuxDownloads/DownloadedISOfiles/Sabayon/Sabayon_Linux_5.1-r1_x86_GAMING.iso
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[#1 SIZE:73.5MiB/3,733.5MiB(1%) CN:11 SPD:16.78KiB/s ETA:62h01m42s]
2010-03-29 18:59:44.463540 NOTICE - Shutdown sequence commencing... Press Ctrl-C again for emergency shutdown.
[#1 SIZE:73.5MiB/3,733.5MiB(1%) CN:0 SPD:0.00KiB/s]
Download Results:
gid|stat|avg speed |path/URI
===+====+===========+==========================================================
1|INPR| 16.9KiB/s|/media/Elements/LinuxDownloads/DownloadedISOfiles/Sabayon/Sabayon_Linux_5.1-r1_x86_GAMING.iso
Status Legend:
(INPR):download in-progress.
aria2 will resume download if the transfer is restarted.
If there are any errors, then see the log file. See '-l' option in help/man page for details.
real 86m11.026s
user 0m1.660s
sys 0m4.824s
paul@paul-ArtistX07:/media/Elements/LinuxDownloads/DownloadedISOfiles/Sabayon$ LS -L
Stopping the process downloading from the torrent and starting the process downloading from a mirror
[#1 SIZE:222.5MiB/3,733.5MiB(5%) CN:25 SPD:16.86KiB/s UP:0.00KiB/s(656.0KiB) ETA:59h15m04s]^C
2010-03-30 00:08:20.810013 NOTICE - Shutdown sequence commencing... Press Ctrl-C again for emergency shutdown.
[#1 SIZE:222.5MiB/3,733.5MiB(5%) CN:0 SPD:0.00KiB/s UP:0.00KiB/s(656.0KiB)]
2010-03-30 00:08:25.991275 NOTICE - Emergency shutdown sequence commencing...
Download Results:
gid|stat|avg speed |path/URI
===+====+===========+==========================================================
1|INPR| 10.0KiB/s|/media/Elements/LinuxDownloads/DownloadedISOfiles/Sabayon/Sabayon_Linux_5.1-r1_x86_GAMING.iso
Status Legend:
(INPR):download in-progress.
aria2 will resume download if the transfer is restarted.
If there are any errors, then see the log file. See '-l' option in help/man page for details.
real 290m10.385s
user 0m6.644s
sys 0m1.544s
paul@paul-ArtistX07:/media/Elements/LinuxDownloads/DownloadedISOfiles/Sabayon$ date && time aria2c --dir=/media/Elements/LinuxDownloads/DownloadedISOfiles/Sabayon http://147.32.127.222/MIRRORS/sabayon/iso/Sabayon_Linux_5.1-r1_x86_GAMING.iso --allow-piece-length-change=true [note : this is necessary to prevent an error from aborting the re-start]
Tue Mar 30 00:09:09 NZDT 2010
2010-03-30 00:09:09.282579 WARN - You may encounter the certificate verification error with HTTPS server. See --ca-certificate and --check-certificate option.
*** Download Progress Summary as of Tue Mar 30 00:10:09 2010 ***
==============================================================================================================================================================================================
[#1 SIZE:231.0MiB/3,733.5MiB(6%) CN:5 SPD:616.26KiB/s ETA:1h37m00s]
FILE: /media/Elements/LinuxDownloads/DownloadedISOfiles/Sabayon/Sabayon_Linux_5.1-r1_x86_GAMING.iso
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
*** Download Progress Summary as of Tue Mar 30 00:11:09 2010 ***
==============================================================================================================================================================================================
[#1 SIZE:267.0MiB/3,733.5MiB(7%) CN:5 SPD:599.03KiB/s ETA:1h38m45s]
FILE: /media/Elements/LinuxDownloads/DownloadedISOfiles/Sabayon/Sabayon_Linux_5.1-r1_x86_GAMING.iso
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
*** Download Progress Summary as of Tue Mar 30 00:12:10 2010 ***
==============================================================================================================================================================================================
[#1 SIZE:302.1MiB/3,733.5MiB(8%) CN:5 SPD:604.49KiB/s ETA:1h36m52s]
FILE: /media/Elements/LinuxDownloads/DownloadedISOfiles/Sabayon/Sabayon_Linux_5.1-r1_x86_GAMING.iso
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
From near the end to the finish
*** Download Progress Summary as of Tue Mar 30 01:45:42 2010 ***
==============================================================================================================================================================================================
[#1 SIZE:3,725.6MiB/3,733.5MiB(99%) CN:5 SPD:304.12KiB/s ETA:26s]
FILE: /media/Elements/LinuxDownloads/DownloadedISOfiles/Sabayon/Sabayon_Linux_5.1-r1_x86_GAMING.iso
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[#1 SIZE:3,733.4MiB/3,733.5MiB(99%) CN:3 SPD:71.92KiB/s ETA:02s]
2010-03-30 01:46:03.826240 NOTICE - Download complete: /media/Elements/LinuxDownloads/DownloadedISOfiles/Sabayon/Sabayon_Linux_5.1-r1_x86_GAMING.iso
Download Results:
gid|stat|avg speed |path/URI
===+====+===========+==========================================================
1| OK| 622.7KiB/s|/media/Elements/LinuxDownloads/DownloadedISOfiles/Sabayon/Sabayon_Linux_5.1-r1_x86_GAMING.iso
Status Legend:
(OK):download completed.
real 96m54.617s
user 0m18.529s
sys 0m43.407s
paul@paul-ArtistX07:/media/Elements/LinuxDownloads/DownloadedISOfiles/Sabayon$ aria2c
In Summary
I think that aria2 has potential for securing a good download of a large file, saving the need to re-download a corrupted file and salvaging any aborted attempts to download the file.
I would like to investigate a web-based application that would use this, plus zsync, to make it easy for anyone to download an ISO file and re-use an existing file that contains a significant portion of the target file.
Paul
PS This is useful information to have in tandem with the above : --
As a way to avoid having to upload the entire ISO to a web server before distributing it, which causes delays and the possibility of a corrupted file being transferred, it might be a good strategy just to start with a torrent.
How to make and host a .torrent file : --
Brian's BitTorrent FAQ and Guide - creating a new torrent
Further Examples.
Download XthumbLive CD with aria2 and Metalinks
Here is a torrent file for XthumbLive from cosbear, posted on Linux Internationals Forum, here.
And here is a metalinks file including both the above torrent and the main server download from http://www.elkspirit.net/XthumbLive.iso .
Download with aria2 using the above metalink, by using this in a Konsole window : --
[code]aria2c -c http://pgtips91.pbworks.com/f/XthumbLive.iso.metalink[/code]
Example of a Metalinks File.
Here is what is in the metalinks file : --
<?xml version="1.0" encoding="utf-8"?>
<metalink version="3.0" generator="Metalink Generator v1.00.0037" xmlns="http://www.metalinker.org/">
<publisher>
<name>cosbear</name>
<url>http://linuxinternationals.org/phpBB3/viewtopic.php?f=104&t=3234&p=25001#p25000</url>
</publisher>
<copyright>GPL</copyright>
<description>XThumbLive CD Installs persistent Xubuntu to a flash drive</description>
<files>
<file name="XthumbLive.iso">
<version>1.0</version>
<language>EN</language>
<os>i386</os>
<size>619708416</size>
<verification>
<hash type="md5">ba35c437168c6b87093a1665f94dfdcf</hash>
</verification>
<resources>
<url type="bittorrent" location="US" preference="100">http://pgtips91.pbworks.com/f/XthumbLive.iso.torrent</url>
<url type="http" location="US" preference="100">http://www.elkspirit.net/XthumbLive.iso</url>
</resources>
</file>
</files>
</metalink>
Comments (0)
You don't have permission to comment on this page.