Tunnelling Your Way Through the Government’s Metadata Retention Laws

If you happen to be fortunate enough to have access to a Unix based web server (such as an EC2 or Linode) machine) you will surprised how easy it is to safely browse the web – circumventing the Australian government’s 2 year mandatory metadata retention laws.

t_29_0You can use an SSH tunnel to use your off-site server as SOCKS Host. A SOCKS Host (or Server) is a general purpose proxy server that establishes a TCP connection to another server on behalf of a client, then routes all the traffic back and forth between the client and the server. It works for any kind of network protocol on any port. Because the connection is secure, only the client and the host can access the the data.

This is how is circumvents Government spying. The only connection your ISP can see is the initial tunnel made to the server. All the delivery of websites etc through that connection are invisible to them (and the government).

It’s tragically easy to setup, simple initiate a SSH connection with dynamic application-level port forwarding, like this:

ssh -D 12345 user@your.server.com

And then, tell your browser that you want to use a HTTP SOCKS 5 Host for proxying (Options > Advanced > Connection Settings for Firefox):

Screen Shot 2015-11-03 at 1.52.46 PMUse manual proxy configuration; set it to SOCKS v5 on the same port you specified as forwarding.

Be warned however, while your Internet traffic will be encrypted and invisible, your DNS lookups will still be public. Firefox has a setting called ‘network.proxy.socks_remote_dns’ which you can enable by browsing to the address ‘about:config’ and searching for the string above.

Lastly, be warned that browsing this way will slightly decrease speed of your browsing – but this may be a small price to pay, and may not even be noticeable.

While this is all trivial for Linux and OSX users; Windows users will need to jump through a few more hoops. This blog post inspired and references an excellent Linode Guide which covers things in more detail, and includes instructions for Windows users using Putty.

Better Way to Copy a Large Quantity of Data Over a Network Without Using ‘scp’

If you ever need to copy a large amount of data over a network (especially if its a huge number of small files) you can pipe a tar command through a ssh connection, and because tar copies whole blocks at a time, it will be far, far, faster than using SCP.

To execute it, simply:

$ tar czf - <files> | ssh user@host "cd /wherever; tar xvzf -"

 

High Performance & Multi-threaded SCP Using RSYNC

Recently, I had the somewhat laborious job to backup a stack of websites and blogs from my remote web server.  Initially I tried do it with a simple SCP command; but after letting it run for about an hour, it was obvious that it was just too slow and taking too long downloading each file one at a time.

After talking complaining to a friend he suggested using the RSYNC command. Surprisingly it was incredibly very easy to get it working, you simply issue:

 rsync -avz -e ssh remoteuser@remotehost:/remote/dir /this/dir/

Obviously changing the appropriate parts for your case.

I found it to be at least 10 fold faster (or more) than SCP on it’s own, and better still, RSYNC will resume when SCP wont!  Try it and see for yourself.

Connecting to Postgres Through a SSH Connection

Production level servers can be tricky.  Mostly because of the inherent, increased security needed to protect online (and exposed?) assets.  Our Postgres server is one such asset.

But opening the necessary ports to allow access to the database server (for maintenance) can expose the server to possible attack.  Fortunately, if your Postgres server is on a UNIX or Linux based machine with SSH installed, you can forward the Postgres port through the SSH tunnel.

ssh -L 1234:localhost:5432 username@server.dns.addressorip.com

In PGAdmin, simply connect to localhost:1234, and you will connect to the production server through the SSH connection without the need to open a port on the external firewall.