Some more tips and tricks

[wirespot]

I've had the WL-500gP for about a week now, using Oleg's firmware, with an USB HDD rack and it works really great. I wanted to point out some stuff I've learned the hard way.

Bittorrent client:

I've read this a lot of times and I didn't believe it until I tried and convinced myself: Oleg's torrent.cgi or transmission.cgi is really the best all around torrent client out there at this time.

As far as console clients go, I've tried transmission, ctorrent, enhanced-ctorrent, tornado and rtorrent. Most of them are single-torrent. tornado is multi-torrent but is not interactive. Really, the best console client out there is rtorrent, but after Oleg managed to port it I suddenly realised it lacks automated queue control and that I much more enjoy a web interface, since it's more likely to have a web browser handy wherever I go than a ssh client.

I've also tried various web interfaces. I've managed to make torrentflux-b4rt work with lighttpd, PHP4 and sqlite2, using tornado as a client (see this tutorial) and then managed to compile the special transmission version that comes with b4rt. With tornado it's a real hog. The combination of PHP, lighttpd, sqlite and Python is simply too much. Simply reloading a PHP page works the HDD a lot. With only one torrent going the load stays above 1.4 and I've heard that once you go to about 4 torrents at once the router is pretty much dedicated to that. I've tried and the load shot to 6 or 7. With transmission it's quite alright, but when I stood back and looked at the thing I realized that it didn't do anything that Oleg's cgi doesn't essentially do already.

And compare it to Oleg's cgi, which gives me average load of 0.3 or 0.4 with 6 torrents going. The only gripe I have with the transmission.cgi is that it doesn't move torrents around the queues by itself (LE yes it does, I just hadn't understood about the watchdog at the time). Well, overall is really a simple and very good torrent client and if I have the time I may hack it one day to clean up the code in the watchdog and cgi and add some more features. (I'd like it to know how to queue more than one torrent at once, or to change the speed limits from the web interface.)

How to allow SSH and other stuff from outside:

If you run iptables -L INPUT on your router, you'll probably see something like this in Oleg's latest firmware (1.9.2.7-7f):

logdrop  all -- anywhere anywhere state INVALID
ACCEPT   all -- anywhere anywhere state RELATED,ESTABLISHED
ACCEPT   all -- anywhere anywhere state NEW
ACCEPT   all -- anywhere anywhere state NEW
SECURITY all -- anywhere anywhere state NEW
ACCEPT   udp -- anywhere anywhere udp spt:bootps dpt:bootpc
ACCEPT   tcp -- anywhere anywhere tcp dpt:ftp
logdrop  all -- anywhere anywhere

The reason many of you haven't been able to open up your outside ports (for SSH and other stuff) is because of that first rule. I don't know what the INVALID chain is for and where it comes from, but that's where the packets stop.

So this is how my /usr/local/sbin/post-firewall looks like:

#!/bin/sh
## FIREWALL
## set default policy
iptables -P INPUT DROP
## deny ftp access from WAN
iptables -I INPUT 1 -p tcp -i "$1" --syn --dport 21 -j DROP
## Allow access to various router services from WAN
for P in 22 65534 8008; do
  iptables -I INPUT 1 -p tcp --syn -i "$1" --dport $P -j ACCEPT
done

What I do with the firewall is:
A. I change the default policy from ACCEPT to DROP, which is saner, security-wise.
B. I insert some rules before the INVALID rule, thus bypassing it: first I deny FTP SYN packets on the outside interface, which cuts access to FTP. Second, I make a list of ports (ssh, torrent and transmission.cgi) and let SYN packages to pass through to them. That's it.

Some more security ideas:

1. For SSH: it would be a good idea to change the default port from 22 to something else, like 24. It cuts down on automatic scanners figuring out you have SSH open. It is NOT a foolproof security measure, but it's good nonetheless. You do this by finding where you start dropbear (probably /usr/local/sbin/post-boot) and adding -p 24 as parameter. Don't forget to change the firewall rule above too.

2. For torrent, opening up 65534 has really helped me. I get a LOT more peers now, about 2-4 times as many as before. I get so many it's ridiculous, it makes some instances of transmissiond have a hard time keeping up with them and use up CPU. (A configurable connection limit in transmissiond would be really sweet BTW.)

3. If you open up web access to transmission.cgi from outside be sure to protect it with a password. See the /opt/etc/httpd.conf help or /opt/etc/init.d/S80busybox_httpd for instructions on how to do that. EVEN SO, it still not good enough! The user and password pass around in cleartext! It will stop someone who accidentally stumbles across your web interface from using it, but if you access it from work for instance almost anyone (your admin definitely) can eavesdrop and pick up the user and password without a problem. So be warned. What would really help here is SSL, but I've yet to figure how to make busybox make a self-signed SSL website.

Wondershaper:

The place to add this is also in /usr/local/bin/post-firewall, at the bottom:

## WONDERSHAPER
/sbin/wshaper start "$1" 400 190

Replace 400 and 190 with appropriate numbers for your connection.

How to determine them? First, you take the maximum figures for your connection. If you know what they're supposed to be from your ISP (like mine are supposed to be 512 and 192) than start from there. Even better, do a test. I personally love the Speakeasy test. It will give you real figures (remember to stop all other traffic, such as torrents, while you do the test!)

So, you got 512 and 192. But for wshaper to work best you actually need to lower them somewhat. How much? To see how much, you need to ping an outside host, preferably from the router console, but you can do it from a LAN computer too. You can ping anything, such as www.google.com.

But the best would be to ping your gateway. To find out what it is, do this command: cat /proc/net/arp. You'll see something like this:

IP address       HW type     Flags       HW address            Mask     Device
89.xxx.xxx.xxx     0x1         0x2         XX:XX:XX:XX:F5:4B     *        vlan1

That IP on the vlan1 line is the gateway, ping that.

So now you have one console where a ping runs, and you watch the times. It would be best to start some heavy traffic, like torrents, too, to see how it does under real heavy conditions. My ping would go to around 300ms under these circumstances. Now, in another router console, run that wshaper command:

/sbin/wshaper start vlan1 512 192

(Note I use vlan1, not "$1", since we're in the console not post-firewall). Then look at the ping and see if it goes lower. It probably won't, not much, so start lowering both the download and the upload numbers slightly and run the command again, then look at the ping again. The download number needs to go down a lot more than the upload. I finally ended up with 400 and 190, and at that point my ping went between 10-60ms.

Note that this WILL NOT LIMIT your download speed! So don't be worried. All it does is to tweak the wondershaper for better latency, it has no connection to the actual download speed you get.

That's it for now, will get back if I find out more worthy stuff.

[oleo]

The only gripe I have with the transmission.cgi is that it doesn't move torrents around the queues by itself

Not true. Transmisison and Torrent move packages from queue to queue from day one.

[wirespot]

Then it means I haven't figured it out. How do they determine how to do that? I've placed torrents under source/ and they stayed queued. I let them be and they wouldn't start by themselves. I had to push them by hand. When they finish they'd stay seeding (I put AUTOSEED=NO) instead of stopping and letting another one from the queue start.

[oleo]

watchdog shoul be run by cron every half an hour

[wirespot]

Watchdog is in cron, of course, and this still happens. I've figured watchdog must be taking care of this but even if I run it manually it doesn't move them around.

[oleo]

watchdog enqueues new if none in download queue (empty).

[wirespot]

Another great use for the router is ripping music from Internet radios.

1. Install streamripper and some additional music format libraries:

ipkg install streamripper libogg libvorbis libmad

2. Choose a directory on your USB HDD or flash disk, let's say /opt/files/streamripper.

3. Start ripping (using the screen utility is probably a good idea):

streamripper http://some.radio.ip -d /opt/files/streamripper

That's pretty much it. Streamripper will rip music, make the necessary subdirectories (incomplete pieces go their own subdir, complete pieces go to a dir named after the radio station) and so on. Look at the other parameters to see what else it can do. Some ideas:
* It can rip in one big file or separate files; but be warned that sometimes stations delay the name of the song on purpose, so it will cut the songs wrong. You may want to use one big file and use an audio editor to cut them by hand.
* It will rip Ogg, MP3 and AAC stations.
* It can relay while it's ripping on a port of your choice, so you can listen to the music from the LAN without having to download the stream twice (or more).

Here's a shell script I use so that I don't have to call streamripper with all my favorite parameters each time. I called it "rip" and I put it in /opt/bin. If you put it under /usr/local/bin remember to save to flash!

#!/bin/sh
[ -z "$1" ] && { echo "Usage: `basename \"$0\"` URL [options]"; exit 1; }
URL="$1"
shift
streamripper "$URL" -d /opt/files/streamripper "$@"

Change the target dir to your own.

If you want Internet radio stations see www.shoutcast.com where you'll find a few thousands to pick from.

This is great use for your bandwidth if you're paying for it anyway and you don't use when not at home. If you rip an AAC stream the files are very small (yet good quality) so 1 or 2 GB of space will give you a LOT of music. And it's perfectly legal to do this according to most countries' copyright law, as long as you do not redistribute the music too much. (not so sure about USA though ).

If you want to rip MMS streams you can use mmsrip. Installation tutorial here (thanks to Mokake).

[wirespot]

I managed to protect the transmission.cgi by SSL. As I said above, not having SSL is not really secure, and even a self-signed SSL certificate is better than sending the user+pass around in cleartext.

It turns out lighttpd supports SSL. So go ahead and install that and openssl: ipkg install lighttpd openssl.

I assume you already have a working setup with transmission.cgi (or torrent.cgi) and busybox_httpd. The CGI is most likely under /opt/share/www/cgi-bin.

1. Kill off busybox_httpd. This means:
* killall busybox_httpd
* make sure it doesn't start at startup (edit /usr/local/sbin/post-boot if that's where you're starting it, or edit /opt/etc/init.d/S80buysbox_httpd and add exit 0 right after #!/bin/sh).

2. Make a new configuration for lighttpd. Go to /opt/etc/lighttpd and copy lighttpd.conf to something else, like transmission-ssl.conf. Edit that and do the following:
* Under server.modules, make sure you uncomment mod_access, mod_cgi and mod_accesslog.
* Change server.document-root to point at "/opt/share/www".
* Look up server.port and make sure it points to whatever port busybox_httpd used to point at (8008, 8081, whatever).
* Uncomment the cgi.assign section. Change the assignment for ".cgi" from "/usr/bin/perl" to "/bin/sh".
* Look up ssl.engine and uncomment it (make sure it says "enable" too).
* Look up ssl.pemfile and point it to a suitable location. I chose "/opt/var/run/lighttpd.pem".

3. You need to create an SSL certificate. You do it like this:

cd /opt/var/run
openssl req -new -x509 -keyout lighttpd.pem \
-out lighttpd.pem -days 3650 -nodes

Notice how I went to the dir I picked in the config file, and how I named the file. I also chose the certificate to work for 10 years (3650 days); since it's self-signed you'll get PLENTY of warnings about it anyway so one more warning about it being expired is overkill.

You'll be asked all kinds of questions about your country, town and so on. I suggest putting bogus information in there, otherwise someone could infer your identity by examining the certificate (all that data will be available freely if you open the transmission cgi interface to the Internet).

4. Edit /opt/etc/init.d/S80lighttpd and edit the location of the conf file. Remember, we changed it to transmission-ssl.conf!

5. Start lighttpd by running /opt/etc/init.d/S80lighttpd start. If everything went fine you should be able to access the interface at:

https://your.ip:port/cgi-bin/transmission.cgi

Note that it's httpS, not simple http!

6. Add the above command at the end of /usr/local/sbin/post-boot. Teoretically /opt/etc/init.d/rc.unslung should take care of it, but in practice it seems to not do so.

7. Save the changes to the post- script by running this!

flashfs save && flashfs commit && flashfs enable

[wirespot]

A lot of people, myself included, are probably having some trouble when they first run into Oleg's web interface for torrent download (transmission.cgi). That's because it's a little different from the torrent programs they're used to. So I'm going to describe what's going on under the hood.

There are 3 pieces of software working on the torrents:
1) transmissiond. This is the actual workhorse that's downloading and uploading a torrent. If you run "ps" in a console you will see several of them working (if you have torrents running). Every time a torrent starts, one of these comes along and takes charge of it.
2) transmission_watchdog. This one runs periodically, usually every 30 minutes, and is in charge of moving torrents around the queues (from "queued" to "active" to "done").
3) transmission.cgi this is the actual web interface and is the thing you use to see your torrents and control them.

How to use the torrents

You start by putting a torrent file under the source/ directory. You can do this in many ways: you can download it on a desktop computer and FTP or SCP it to the router; you can SSH to the router, go into that directory and wget it from a website; you can use the "Fetch" button in the web interface to give the URL and the cgi will wget it and put it under source/.

All torrents found under source/ are automatically seen as part of the "Queued" set of torrents. If there are no torrents in the "Active" set, the watchdog will fetch the first one in the Queued set the next time it runs and make it Active. Or, you can manually make as many torrents as you want Active by selecting them and using the Push button in the interface.

Everytime the watchdog runs it will also check for completed Active torrents (the ones seeding) and will move them to the Completed set, where they stop moving and wait for you to come get the files.

You can manually pause an Active torrent by selecting it and using the Push button. This will stop the torrent and put it in the Suspended set. It will not leave this set anymore unless you select it and use the Push button again. (That's right, Push does two different things; when the torrent is first Queued it moves it to Active, and then it moves it between Active and Suspended; there's no way to move it back to Queued anymore).

Active torrents (and their files) are found under the work/ dir. Queued torrents are under source/. Completed torrents are under target/. Suspended torrents are under work/, but are temporarily renamed to ".torrent.suspended".

What other buttons do:

List: this one gives the complete list of all the torrents in all the queues.

Update: this one updates the Active information with actual data. Since this update is an expensive operation for the router, it is usually cached and you need to press Update to refresh it. Otherwise, using List will only show the same old info.

Watchdog: this will force the watchdog to run now instead of waiting for the next normal time. This is useful if you see that a torrent has finished downloading and is seeding, but you don't want to wait for it to be moved to Completed the next time the watchdog would normally run. (Please note that you should let torrents seed so other people can benefit too. Instead of stopping that torrent, consider Push-ing a Queued one instead.)

Pause: this will stop all torrents temporarily. It does this by telling all transmissiond programs to die. It doesn't move torrents around the queues at all, there simply aren't downloads or uploads going. This will be marked clearly in big bold letters saying "torrent processing paused". Use Pause again to resume the Active torrents.

Info will show some information about the torrent (tracker, what files are in it and so on).

Log will try to create a graph showing the state of the downloads. Depends on whether you have gnuplot installed. (I personally prefer rrdtool, look around the forum for the rrdtool tutorial.)

Remove will mark a torrent for removal, and Purge will actually DELETE BOTH THE TORRENT AND THE FILES FOR IT. So be careful with it.

Scrape will attempt to ask the tracker for the scrape info, and Best will attempt to suggest the best seed torrent from the completed ones.

Note will add your personal note to the torrent, in case you want to jot down something about it.

[wirespot]

User+password access to your webserver is very very useful because, together with SSL encryption of the connection, it allows you to access your router fairly safely even when not at home.

Warnings:
* SSL by itself is useful, but not much use if anybody can access the router webpages without being asked for user+password.
* This is NOT ABOUT PROTECTING THE SETUP interface. I'm only talking about protecting websites you set up on the router by yourself, such as the transmission cgi interface. The web setup of the router is another breed of webserver who runs separately.
* Even when using SSL+user+pass, you ARE NOT COMPLETELY SAFE. The owner or admin of the computer you're using may have installed keyloggers or network transparent proxies which will intercept your user+pass or decode SSL. They protect against casual eavesdropping, but will not stop a determined and knowledgeable person with access to your immediate network setup or to the computer you're using. Nothing can do that.

These being said, here's how to add user+pass to lighttpd:

NOTE: I've chosen digest authentication because it is a bit more secure than basic authentication. If you want basic change "digest" to "basic" and "htdigest" to "htpasswd" in the instructions below. Be WARNED that dumber browsers (like lynx) cannot use digest authentication!

1) Install lighttpd: ipkg install lighttpd.

2) Copy the default config file (/opt/etc/lighttpd/lighttpd.conf) to another file. This is so you don't risk it being overwritten by mistake during upgrades.
NOTE: if you took the SSL tutorial above and already have an SSL-enabled config, very good, modify that to add user+pass!

3) Edit the config file. Under server_modules, make sure to uncomment "mod_auth".

4) Scroll down and look for "auth.". Set auth.backend to "htdigest". Set auth.backend.htdigest.userfile to the a file where you want to keep users and passwords. Example: /opt/etc/lighttpd/.passwd.

5) Modify the auth.require section. Make sure it looks something like this:

auth.require = ( "/" =>
  (
    "method"  => "digest",
    "realm"   => "ABC",
    "require" => "valid-user"
  )
)

Make sure to close all the brackets, and watch out for those commas!

The "/" is the directory you want to protect, relative to the webserver root dir. Here I say I want to protect all the files.

The method is "digest", and because I chose digest I have called the auth. directives above accordingly! Have I called it "basic", for example, I'd have changed the auth. stuff too.

Realm is the name of the authentication set. Remember what you write here, we'll need it in a moment below. (Every bracket set needs its own name, because other things may be the same across more than one set and can't be used to refer to them.)

Finally, require means who you want to be able to authenticate. "Valid-user" means any user that supplies a valid user+pass. You could also say "user=john" and only allow john in that set, even though mary also has a valid user+pass.

6) You're done with the config, save it and restart lighttpd whenever you want.

7) Now you need to generate the passwd file. You can't just edit it by hand. You need the "htdigest" program, which is found in the apache package. You can "ipkg install apache", of course, but I prefer to go to a temp dir (NOT /tmp!), do "ipkg download apache" and then look inside for what I want (the ipk files are actually tar.gz files and have another "data.tar.gz" file in them).

8) One way or the other, you got htdigest. Use it whenever you want to generate it or change a password or add a user:

#to generate it:
htdigest -c /opt/etc/lighttpd/.passwd ABC john
# change john's password for the ABC realm
htdigest /opt/etc/lighttpd/.passwd ABC john
# add mary on the XYZ realm
htdigest /opt/etc/lighttpd/.passwd XYZ mary

When you want to delete users edit the files with a text editor and delete lines you don't want.

9) Test your server. If you can't log in, some common errors are:
* wrong user and/or password
* you didn't put the same realm in the conf and the password file
* the password file can't be read (check permissions)
* your browser can't do digest authentication

Check your lighttpd error log (probably /opt/var/log/lighttpd/error.log) for clues.

[wirespot]

I've modified transmission_watchdog and transmission.cgi to accomplish a small feature I wasn't able to live without anymore.

Sometimes I have these torrents that are extremely slow for some reason (no seeds ATM, slow peers etc.) I'd like them to keep on doing their job no matter how slowly, but I don't want them to be counted as running by the watchdog, because they keep the queue occupied and other queued torrents don't get a chance to run.

So what I've done is a 2 step hack:

Step 1: hacked transmission.cgi to offer a button that toggles an "ignored" flag on running torrents.

To do this, edit transmission.cgi (tipically found in /opt/share/www/cgi-bin/).

a) Add the following function somewhere:

# Toggle ignore flag
_toggle_ignore()
{
    if [ -z "$ID" ] ; then
        echo "<b>Please select torrent first!</b>"
        return
    fi
    _find

    if [ -f "${WORK}${TORRENT#${WORK}}" ]; then
       if [ -f "${TORRENT%.torrent}.torrent" ]; then
           DIR=$(dirname "$TORRENT")
           [ -f "${DIR}/ignored" ] && rm -f "${DIR}/ignored" || touch "${DIR}/igno
           echo "<b>$TORRENT toggled</b>"
       fi
    fi
}

b) Find the buttons, near the end of the file, and add this one:

<input type=submit name=ACTION value="Ignore">

c) Find the actions, almost all the way to the bottom, and add this one:

"Ignore") _toggle_ignore ; _list ;;

d) To show the ignored flag, find a place that looks like this:

            if [ -n "${URL}" ]; then
               echo "<td><a href=\"${URL}\" target=_blank>${DUMMY}</a></td>"
            else
               echo "<td>${DUMMY}</td>"
            fi

And change it to look like this:

            if [ -n "${URL}" ]; then
               echo "<td><a href=\"${URL}\" target=_blank>${DUMMY}</a>"
            else
               echo "<td>${DUMMY}"
            fi
            [ -f "$P/ignored" ] && echo "<b style="color:red">[I]</b>"
            echo "</td>"

Step 2: make the watchdog actually respect the flag.

Edit /opt/sbin/transmission_watchdog. Find a place that looks like this:

# Check if any torrent file exists in run env.
TORRENT=`ls -1 $WORK/*/*.torrent 2>/dev/null | head -n 1`

# Start up new  torrent if work is empty
if [ -z "$TORRENT" ]; then
    max_active_torrents=10
    if [ ${ACTIVE_TORRENTS} = ${max_active_torrents} ]; then

And change it to look like this:

WERKIN=$ACTIVE_TORRENTS
IGNEUR=$(ls "$WORK"/*/ignored 2>/dev/null | wc -l)
DIFFER=$((WERKIN-IGNEUR))

echo "<p>${WERKIN} active torrents, ${IGNEUR} ignored, ${DIFFER} acknowledged.</p>
if [ "$DIFFER" -lt 1 ]; then
    if [ "$DIFFER" -gt 9 ]; then

How to use:

When in the cgi web interface, select an Active torrent then click on the "Ignore" button. You should see a bold I appear next to it. Do it again to clear the flag.

You can only toggle the flag for Active torrents. As long as they have the flag, they are not counted towards active torrents when the watchdog determines if it needs to promote more torrents from the Queued ones.

Use this to mark slow torrents which you want to keep chugging along slowly and not keep back the other torrents.

Warning! ipkg upgrade may override the modified files!

Everytime the transmission package is upgraded you may lose the files you've just modified! So you are advised to make copies for personal use and name them something else. To do this:

a) Rename transmission.cgi to transmission2.cgi.
b) Rename transmission_watchdog to transmission_watchdog2.
c) Edit your cgi (transmission2.cgi), look at the bottom, find transmission_watchdog among the actions for the buttons, and replace it with transmission_watchdog2.
d) Start using /cgi-bin/transmission2.cgi instead of the original.

[oleo]

Nice. I was also thinking about that.
I will include this in next transmission package release.

Another wish of mine is to show also upload/download percentage and maybe a ascending listing of best seed suggestion.

[wirespot]

If you include it, please note that you can only toggle the ignore flag on active torrents. If it ends or you push it, for instance, you can't toggle it anymore. So perhaps the check for Active should be skipped, to allow it to be toggled in any state.

[oleo]

Patchlevel 3 in http://trac.nslu2-linux.org/optware/changeset/5505 includes Bypass capability and proper error handling.

[wirespot]

I've hacked wshaper to take a 9th parameter, which specifies ports that are to be given priority over other kinds of traffic.

To hack wshaper yourself, edit it and look for the following two snippets of code. The bold parts are to be added by you, the non-bold parts are what you should look for:

# low priority destination ports
NOPRIOPORTDST="$8"
# high priority destination ports
HIPRIOPORTDST="$9"

    match u8 0x10 0xff at 33 \
    flowid 1:10
 
# some traffic suffers a better fate
for a in $HIPRIOPORTDST
do
       tc filter add dev $DEV parent 1: protocol ip prio 14 32 \
          match ip dport $a 0xffff flowid 1:10
done
 
# some traffic however suffers a worse fate
for a in $NOPRIOPORTDST
 do

I think I did it properly. I've also tested it: after I made use of the new parameter, I've used a bandwidth measurement site while I had a couple of torrents going. I got the full speed of my line in the test, even though the torrents were using about half of it. This is because now port 80 (http) takes priority.

To use this you need to run wshaper at least once on your router, and re-run it after every boot. It's best to edit /usr/local/sbin/post-firewall and add the call to wshaper in there. Remember to save the modifications to flash!

Here's how I run it:

wshaper start $1 400 190 "" "" "" "" "20 21 22 25 110 143 80 443 587 995"

If you run wshaper from post-firewall you need to use $1. If you run it from the command line you need to use "vlan1".

To determine the proper values for the 3rd and 4th parameters (400 190) you need to read the first post in this thread.

The ports I've included in the example are: FTP, SSH, SMTP (outgoing email), POP3 and IMAP (incoming email), regular web, secure web, plus a couple of other ports I'm using.

REMEMBER! This setup refers to ports on remote sites or machines, opened with programs on the computers in your LAN (or on the router).