LINUX GAZETTE
[ Prev ][ Table of Contents ][ Front Page ][ Talkback ][ FAQ ][ Next ]

"Linux Gazette...making Linux just a little more fun!"

Getting Debian: rsync Examples

By

The instructions for using your computer do not always include how to get a free operating system. [That's an understatement! -Ed.] Debian GNU/Linux is a free network operating system. You can get a Debian CD image file (".iso" or ".raw") for free from a download from the Internet. Then you can make a bootable install CD from that ".iso" file. You will need to use the ftp and rsync programs.

When you post a question about how to get a Debian CD image file, you can run into a heavy work day. A heavy work day can often frustrate you because you may get the "speedy fast answer". On those "speeding fast answer days" you would be asked to read the page at cdimage.debian.org. However, you may get stuck. Welcome to you new folks; you may get especially stuck if all of this Open Source freedom of speech and freedom of source code is new to you.

Some "speedy fast answers" may ask you this question: "Do you want to know how to do this?". To the potential new Debian-folks out in cyberspace: read between the lines. You are being offered some clues. Just say yes, my friend. So, for all of you folks who want to know the straight and direct commands, this Bud's for you. This is how you get Debian with rsync:

rsync -rv trumpetti.atm.tut.fi::debian-cd/2.2_rev3/i386/binary-i386-1_NONUS.iso /right/here
To save trumpetti's bandwidth, use a mirror in or near your own country instead. The list of mirrors is at http://cdimage.debian.org/rsync-mirrors.html.

Want to go faster?

You can go blazing fast with costar's pseudo-image kit for making a Debian CD.

  1. *Get a list for your CD
  2. *Make a note of a fast ftp server or even your local mirror of Debian
  3. *Give the list to the pseudo-image kit to make a pseudo-image file
  4. *Rename the pseudo-image file to an official CD image (match your list)
  5. *Use rsync to make it an exact official CD
  6. *Check your CD with md5sum
  1. Get a list for your CD 
    rsync -av trumpetti.atm.tut.fi::debian-cd/2.2_rev3/i386/binary-i386-1_NONUS.list /right/here

http://cdimage.debian.org/cd-images/2.2_rev3/i386/binary-i386-1_NONUS.list
  2. Make a note of a fast ftp server or even your local mirror of Debian.

    You can make a mirror of the Debian archive at home. When you want to access the mirror with the pseudo-image kit, you tell it the path to your mirror and also include the "debian" directory so that there is a "dists" directory waiting underneath.

    /mnt/mirror/debian

ftp://ftp.us.debian.org/debian
  3. Give the list to the pseudo-image kit to make a pseudo-image file 
    ./make-pseudo-image binary-i386-1_NONUS.list /mnt/mirror/debian
--or--
sh make-pseudo-image binary-i386-1_NONUS.list ftp://ftp.us.debian.org/debian
**

    **There must be a dists directory directly under the mirror directory.

  4. Rename the pseudo-image file to an official CD image (match your list) 
    mv pseudo-image binary-i386-1_NONUS.iso
  5. Use rsync to make it an exact official CD 
    rsync -av trumpetti.atm.tut.fi::debian-cd/2.2_rev3/i386/binary-i386-1_NONUS.iso /right/here
  6. Check your CD with md5sum

    There are some crippled MD5SUMS files that are next to useless. Here is a crippled one:

    MD5 (my-fine-new-iso-file.iso) = 9ce8f9fd8f5f1f47efe3eb77d5aae96b

    They have the filename and the sum but the "md5sum -c" option can not be used on these files. Here is the way it should look:

    9ce8f9fd8f5f1f47efe3eb77d5aae96b my-fine-new-iso-file.iso

    There is a space or two between the sum and the filename. Do not put three spaces or a space will be part of the filename.

    
md5sum -c MD5SUMS

Why did I do all of this?

To get it all faster. Like this afternoon.

Elapsed time of a download on the release date from one of few CD iso image sites: Lots of hours and possible interrupted transmission.

Elapsed time of a download on the release date from one of many Debian package mirror sites: Lots fewer hours and less possible interrupted transmissions.

Your CD image will be made via ftp or even direct from your local mirror. Instead of wandering around in ftp-land, you got a list and nabbed only the files you need to make a CD. Smart.

Update your existing CD

To update your CD image, rsync needs the same filename on each end.

Let us say that you have made a dd of your CD.

dd if=/dev/cdrom of=/here/is/mynewfile.iso

You can keep it around and update it with rsync.

You need to know the name of the same CD file on the rsync server.

Then you can rename this new iso to the right name.

Filenames to find

To find the filenames:

Find an rsync server.

rsync -avn the.server.goes.here::

and then its modules will appear.

example:

rsync -avn ftp.fifi.org::

You will see a module named debian or debian-cd and you can infer that debian-cd probably has the iso images you want.

Now get the list inside that cdimages module:

rsync -avn the.server.goes.here::cdimages/

rsync -avn ftp.fifi.org::debian-cd/

will show you every entry in that module. A name and a directory location path for your CD image file will become apparent.

Encryption to get, security it is

The image you want is the non-US version because it comes with security pgp and encryption software. You are not selling it from inside the USA to non-approved entities so you go for it. You will need to find a non-US image.

rsync -rvn the.server.goes.here::cdimages/ | less

rsync -rvn trumpetti.atm.tut.fi::debian-cd/ | less

will get the list and it will be piped into the less pager.

The pager less can go blank while you wait on a delayed loading of a huge list, and you can output to a file:


rsync -avn the.server.goes.here::cdimages/ | less -o my-list-file

rsync -avn ftp.fifi.org::debian-cd/ > my-list-file

example:

The accidental big-big list:

rsync -avn ftp.fifi.org::debian/ > my-new-list-file

If you try to get a whole Debian mirror list of files it can take 7 megabytes to make your list file. Really. There is a whole world of software in Debian.

The fifi site has a module called debian and if you try for a list of files from there you will get a big file indeed.

Rename to match the CD

To get an rsync update, your file has to be named the same.

mv mynewfile.iso /right/here/potato-i386-1_NONUS.iso

example:

The command line length begins to wrap around on you and it can be shortened with the backslash newline character.

rsync -rv --block-size 8192 the.server.goes.here::cdimages/2.2_rev3/i386/potato-i386-1_NONUS.iso /mnt/here/is/where/it/goes

There is a space between "iso" and "/mnt".

rsync -rv --block-size 8192 \
the.server.goes.here::cdimages/2.2_rev3/i386/pot\
ato-i386-1_NONUS.iso /mnt/here/is/where/it/goes

You can chop it up with a backslash to compact the line.

Updating you are, rsync you do


rsync -rv trumpetti.atm.tut.fi::debian-cd/2.2_rev3/i386/binary-i386-1_NONUS.iso /right/here

The above command will put the potato-i386-1_NONUS.iso in the /right/here/ directory.

Only the differences between the two files will be transmitted. You save a lot of bandwidth.

The update for you, a checklist it is

  1. Your file is named the same as the one you want.
  2. You want rsync of that remote file to synchronise with yours.
  3. rsync makes a shadow file of the same size in the rsync temporary directory. Then it moves it to your new filename.

Making a 670 MB file requires 670 MB free space!

For an update that is 670 MB in addition to the CD image file already on your disk.


rsync -av --timeout 999999 --block-size 8192 the.server.goes.here::cdimages/2.2_rev3/i386/potato-i386-1_NONUS.iso /right/here

rsync -av --timeout 999999 --block-size 8192 \
the.server.goes.here::cdimages/2.2_rev3/i386/po\
tato-i386-1_NONUS.iso /right/here

There is a space between "iso" and "/right".

The space has run out, a temp dir we need

You can assign a temporary directory with -T if there is no room on the working directory partition. You need room for the huge shadow file which is the same size as that CD image file you are getting.

type
df
to see your space

rsync -av --timeout 999999 --block-size 8192 -T /my/temp/dir \
the.server.goes.here::cdimages/2.2_rev3/i386/pota\ 
to-i386-1_NONUS.iso /right/here

Caution:

There is a little catastrophe that you can do to yourself:


rsync -av the.server.goes.here::cdimages

will try to dump all of the contents of cdimages onto you!

put the / at the end of the line.

The "-n" option is the "--dry-run" and will show you what it would have done. ("Let's not and say we did.")

rsync -avn the.server.goes.here::cdimages/

will only show the list.

real life example:


rsync -av trumpetti.atm.tut.fi::debian-cd

will get you 16 gigabytes of CD image goodies in all architectures. Enjoy!


rsync -av trumpetti.atm.tut.fi::debian-cd/ | less

shows a big list. Maybe use the "-n" to be safe when you are looking for files.

Big update right now

rsync -av --timeout 999999 --block-size 8192 -T /my/temp/dir \
trumpetti.atm.tut.fi::debian-cd/2.2_rev3/i386/potato-i386-1_NONUS.iso \
/mnt/here/is/where/it/goes

will dump the new CD image file into /mnt/here/is/where/it/goes.

The manual does not have the examples shown here.

The rsync program is ready

You can make a backup, you can use ssh, you can execute commands on the remote to make the file list, you can exclude files, you can include files, you can move the temporary directory, etc. It is a fine copy program and only the differences between files of the same name are transmitted across the bandwidth.

You have got to have the same names of files locally. This rsync does a dandy job of mirroring, but if there are always a ton of new filenames then you are just doing ftp.

Here is a first timer's primer:

  1. First, get to typing. Limber up your fingers, plural. Now we will start with a small exercise. On the console or in an xterm please type the two letters df and then press Enter.
  2. Next, make note of how much free space it tells you about. Each block counted is 1024 bytes. 356784 output from df means three hundred and fifty six megabytes plus seven hundred and eighty-four blocks of 1024 bytes (356784 * 1024 bytes). It is 357 MB in shorthand.
  3. If you have a 670 MB CD file and the extra 670 MB of room for the working shadow file then go ahead and use rsync to update a CD file.

A lot of you are going to use rsync to grab a new CD file. Do us all a favour and keep that original CD file. You can update it to the new version by only getting the differences via rsync.

use rsync to update and backup your sites

And now I am going to criticize everybody and his dog. You folks are wasting bandwidth. Are you always editing and slightly changing your web site? Then use rsync to mirror your web site. Want to add 50k of scripting everywhere on each of your pages? Then keep the same filenames and mirror it with rsync. Did you stay on the cutting edge and download that new iso file and make a brand new CD? Then rename it and update it with rsync for the newest version. What am I getting at?

rsync saves bandwidth

When you try to get a Debian CD the huge list of files will be loaded into memory and that is what is taking so long. It is not stalled. Wait. Have patience. I quote Mr. Miyagi in the Karate Kid: "Wax on. Wax off." Each of those many files is being opened for a checksum to see if it needs to be transmitted. Wait.

Get the md5sum.txt file. When it is in your directory after the rsync, type

md5sum -c md5sum.txt

and the file will be checked for accuracy.

costar's pseudo-image kit

Get the pseudo-image kit here.

Get the list for the pseudo-image kit.

rsync -rv --timeout 999999 --block-size 8192 -T /my/temp/dir \
trumpetti.atm.tut.fi::debian-cd/2.2_rev3/i386/binary-i386-1_NONUS.list \
/mnt/here/is/where/it/goes

will put the list file in /mnt/here/is/where/it/goes. It is called binary-i386-1_NONUS.list.

That binary-i386-1_NONUS.list will be used to build your CD image. If you have a partial mirror of Debian (oh sure, 14 gigabytes of stuff) then the pseudo-image kit will make a CD for you from your local mirror. It is very blazing fast for all you speed freaks. I must admit with disk drive prices being reasonable it is time for all of us to buy up the remaining stock of usable drives. You may not know about the effort to lock you down, but the age of freedom of fair use is ending.

Why not use ftp?

Why not slam your ftp server with 670 more megabytes and several hours of work?

To quote the clear coder J.A. Bezemer from a thread on debian-cd:

"
> Most people get bits from the local hard disk a lot faster than
> they get bits over the network.  The pseudo-image kit finishes in
> minutes, instead of hours, on most "fast" connections.

That's one part of the story; the other is that we have only a few CD image
mirrors and about 250 packages mirrors, and the Kit does a great job to
distribute the used bandwidth more evenly between all mirrors. Remember that
every single bit flowing from any of the Debian mirrors is sponsored by
someone, and this way the people who invested in big disks for the CD images
don't have to pay that much for their net connection.
"
'Nuff said.
Using one finger over and over on the clicker button will give you a pain.
Read means look at every word and try to make sense of it. It does not mean look for an easy link where someone does it for you and then give up. Capital Letters count. Punctuation counts. Poor writing can confuse, so have fun by pointing out the error to someone and then we can all fall all over ourselves to get the answer posted. It is a weird world. Use it to your advantage.
You get to the virtual console from X windows by pressing Control + Alt + F2. It takes two hands and you may be surprised to learn that you can use Linux without X. You can even install it without a graphical user interface. You can even use Linux with 16 megabytes of RAM on these modern distributions. But not Mandrake; it will complain of low resource unless you have 64 MB RAM and a Pentium.
Make a partial mirror of Debian by using one of the contributed scripts from the Debian-folk. They make use of rsync and ftp for cutting down the size of the archive to suit your particular interest. The idea is to match the archive to the architecture of your machine.
Your next "advance" in hard disk storage is the copy-protected drive. It means you will not be able to use everything in the world any more. You lose each freedom by degrees. At some point you must revolt or become revolting.

The copyrighted material on a music CD is definitely covered with legal rights and permissions. With the genetically engineered mutation of Napster, we can clearly see the lock-down coming. New hard disk drives and other storage mediums are in danger of being copy-protected. You can use software, but it has to be approved software. Sounds all warm and fuzzy, eh? Put it this way: Thou shalt not reproduce.

To quote the clear coder J. A. Bezemer from a thread on debian-cd: 
"Using --partial will kill your valuable pseudo-image if rsync fails after 1
byte has been transferred. So either back up your pseudo-image or do NOT use
--partial."
So do not use partial, eh?

Reference reading:

rsync.samba.org -- the home of rsync and its Faq-O-Matic

NSBD -- Not-So-Bad-Distribution at Bell Labs uses rsync to securely update you

rsync resources -- Multiple system install and updates with rsync

rsync resources -- Setup anonymous rsync servers

Help File for Captives -- if you are seeking freedom


made with mcedit on an i486 with GNU/Linux 2.0.35

There were no freeze-ups or crashes during testing of these procedures. We suspect GNU/Linux software is to blame.

Bill Bennet

Bill, the ComputerHelperGuy, lives in Selkirk, Manitoba, Canada; the "Catfish Capitol of North America" if not the world. He is on the Internet at www.chguy.net. He tells us "I have been a PC user since 1983 when I got my start as a Radio Shack manager. After five years in the trenches, I went into business for myself. Now happily divorced from reality, I live next to my Linux box and sell and support GPL distributions of all major Linux flavours. I was a beta tester for the PC version of Playmaker Football and I play `pentium-required' games on the i486. I want to help Linux become a great success in the gaming world, since that will be how Linux will take over the desktop from DOS." It's hard to believe that his five years of university was only good for fostering creative writing skills.

Copyright © 2001, Bill Bennet.
Copying license http://www.linuxgazette.com/copying.html
Published in Issue 67 of Linux Gazette, June 2001

[ Prev ][ Table of Contents ][ Front Page ][ Talkback ][ FAQ ][ Next ]