OSX

Better Way to Copy a Large Quantity of Data Over a Network Without Using ‘scp’

Better Way to Copy a Large Quantity of Data Over a Network Without Using ‘scp’

If you ever need to copy a large amount of data over a network (especially if its a huge number of small files) you can pipe a tar command through a ssh connection, and because tar copies whole blocks at a time, it will be far, far, faster than using SCP.

To execute it, simply:

$ tar czf - <files> | ssh user@host "cd /wherever; tar xvzf -"

 

How to Speed Up Importing or Recovering a Large MySQL Database

How to Speed Up Importing or Recovering a Large MySQL Database

I recently had to import a 30GB MySQL database from a backup of a client's production database. My development workstation really struggled with the hundred's of thousands of INSERT queries, and the import either took an unacceptably long time or failed outright.

Fortunately, there is a sure-fire way to increase the import though (in my case a 100 fold increase in speed).

Simply open Terminal and type:

> mysql -uXXX -pXXX

...and replace the X's with an appropriate username and password and then paste this at the MySQL prompt:

CREATE DATABASE my_database;
USE my_database;
set global net_buffer_length=1000000;
set global max_allowed_packet=1000000000;
SET autocommit=0;
SET unique_checks=0;
SET foreign_key_checks=0;
SOURCE /some/path/database_file.sql;
COMMIT;
SET autocommit=1;
SET unique_checks=1;
SET foreign_key_checks=1;

A few notes: 'set global net_buffer_length' and 'set global max_allowed_packet' only apply if your source filename and path are over a network. Also simply omit the 'CREATE DATABASE my_database;' line if you already imported the blank schema, or the database already exists.

Monitoring the Progress of a Very Large MySQL Database Import

Monitoring the Progress of a Very Large MySQL Database Import

Previously I have mentioned an awesome little command-line tool called 'pv'. Recently, I was trying to restore a clients of legacy database on my development machine, which was a staggering 30GB .sql file. I was having quite a few problems trying to wrestle this beast, the worst of which being that I had no idea how much longer the import would take, or if it had locked up. Luckily, PV comes to the rescue.

Normally, when importing a MySQL dump file, you can just type:

mysql -uxxx -pxxx dbname < /sqlfile.sql

...to import directly from the file. However, you can pv the file and pipe it into the mysql executable like:

pv sqlfile.sql | mysql -uxxx -pxxxx dbname

And you will get an awesome progress bar about how completed the task is. You can also use it in the reverse. You'd be surprised how useful it can be.

Hands-down, the BEST way to Install MySQL on Mac OSX Mavericks

Hands-down, the BEST way to Install MySQL on Mac OSX Mavericks

I recently had some database woes. I needed to restore a MySQL database of a clients existing website that was dozens of gigabytes in size. I had a great deal of trouble trying to import that data, but that's a story for another time. This was about getting MySQL installed in the first place (for a development environment).

At first, I tried using Homebrew - because its awesome and I like it. But sadly, for one reason or another the default configuration just wasn't working for me. It was simple and blind, but too well hidden and I didn't feel like there was enough "control" (like getting setting/getting the default root password for example). If you're going to automate something, then automate it - but don't ask me to run stuff to secure my install when your supposed to be automating it for me.  Bah humbug!

However, the wonderful folks at macminivault.com solved all my problems. It was simply perfection. Exactly the right balance of automation and control.

Just open up Terminal and paste this into it:

bash <(curl -Ls http://git.io/eUx7rg)

It will tell you or prompt you for the rest. And don't forget to get the text file containing the root password, before you mistakenly delete it.

Simplicity itself.

(comic property of http://www.brainstuck.com/)

How to Fix ‘Requirements installation failed’ When Installing RVM Ruby on OSX Mavericks

How to Fix ‘Requirements installation failed’ When Installing RVM Ruby on OSX Mavericks

Installing Ruby with RVM on Mac is a cinch, simply execute:

\curl -sSL https://get.rvm.io | bash -s stable --ruby

But recently, while trying to install RVM instead of the default Mavericks Ruby, the script that normally just "works" fails with the ominous message:

: Requirements installation failed with status: 1.

It turns out to be something funky with Mavericks and the Homebrew step of the installation. Luckily, despite the failure of the script, you can simply install Hombrew manually to solve the problem. Installing Homebrew is a triviality, simply execute the following command in Terminal:

ruby -e "$(curl -fsSL https://raw.github.com/Homebrew/homebrew/go/install)"

It will first install the XCode Command Line Tools (don't worry if you already have it, just hit "install" and let it do it's thing) and then once the Xcode dialog disappears, hit [any] key in Terminal and it will auto-download and auto-install itself. Once its all finished (and it can take a few minutes, just be patient), simply re-execute the command to install RVM Ruby.

Showing Hidden Files in OSX Finder

Just like when using Windows, sometimes it is necessary to make special hidden system files visible to Finder.  There is no preference for it, but with a simple Terminal command, things can be made visible very easily, on a privileged user account.

Simply open the 'Terminal' application, and at the prompt, type:

  1. defaults write com.apple.finder AppleShowAllFiles TRUE

and then:

  1. killall Finder

This will cause all Finder windows to close and then reopen with the hidden files, visible and identifiable with a ghost-like appearance. You should be able to interact with them normally now.

Screen Shot 2013-11-28 at 11.00.54 pm

Syncing Google Contacts with iMessage (and Contacts) on OSX Mavericks

You may remember earlier in the year I took issue with the super secret hidden checkbox in the Mountain Lion's version of Contacts which magically syncs your Google contacts with OSX Contacts and Messages. As it seems, Apple have finally fixed the SSL problem stopping you from syncing through the Google option in System Preference's Internet Accounts setup.

Getting the sync to work now is as trivial as adding a Google account to your Internet Accounts list, log in and turn on the items you want to sync. Screen Shot 2013-11-26 at 9.46.42 pmHurray! Easy peasy.

It's beyond me that it wasn't like this before.

 

Pipe Viewer (pv) in Mac OSX

 

Update:
Forget what I said.  Install Homebrew (really, do it now) and use HomeBrew to install it.


Pipe Viewer (pv) - is a terminal-based tool for monitoring the progress of data through a pipeline. It can be inserted into any normal pipeline between two processes to give a visual indication of how quickly data is passing through, how long it has taken, how near to completion it is, and an estimate of how long it will be until completion. However, it isn't included by default in Mac OSX.

The good news is that there are several ports of pv, you just need to go and grab one:

  • HomeBrew: Run "brew install pv" to get the latest version.
  • MacPorts: Run "port install pv" to get the latest version.
  • Or (recommended) install the Rudix pv port (a simple package installer)

Pv allows you to get a really awesome progress of your terminal commands to see how things are going; especially useful for long operations (such as cp or tar etc) so you know everything is ticking over time (and perhaps even giving an ETA for completion):

13.2GiB 1:33:17 [3.57MiB/s] [================================>                ] 67% ETA 0:44:4

I highly recommend this for anyone doing long, large or complicated terminal commands.  It's outstanding! To learn more about using Pipe Viewer, this is a great resource.

 

Syncing Google Contacts with iMessage (and Contacts) on OSX Mountain Lion

Being able to sync your Google Contacts with your OSX Contacts is possible - it just takes a bit of digging.

As a long-time iPhone 4 user, I was recently introduced to the wonderful OSX Messages app, which lets me read/send iMessages from my Mac, as if I had sent a text from my phone. Such an amazing thing (even if it does only work with other iPhone users). However, it only used phone numbers to identify contacts; less than ideal since I seldom reference my friends and family by their cell numbers.

I use Gmail for all my email purposes; and after loosing my completely contacts list TWICE from the failures of Windows Mobile 6, I am also a huge advocate for Google Contacts - awesome synergy between my phone list and email contacts. and while I've been very happy Syncing my email/calendar/contacts between Google and my iPhone; my Mac just didn't want to come to the party and I refused to retype my contacts manually.

At first I tried syncing the contacts over Google's Microsoft Exchange Protocol (which is actually the best way of syncing Gmail/contacts/etc onto the iPhone), however you cannot get the necessary SSL to connect to Google on Mountain Lion. Next I tried backing up my iPhone contacts into iCloud, which also failed.

I was about to give up until I discovered that the preferences for the Contacts app has an option that I didn't know about. Carefully disguised under the “On My Mac” account, you will find the option to sync with Google.

syncing-google-contacts-in-mountain-lionHurray! Easy Peasy.

Getting SSI to work in MAMP

It's no viagra buy now secret that I adore my Mac, mostly because it just makes my life easier.  I enjoy the security of OSX, I love it's responsiveness and I love that my operating system doesn't punish me for installing applications and development stacks just to 'play with them'.  A while ago, while looking for a stack that would let me quickly and effortlessly get Apache, MySQL and PHP5 working together without having to muss about with configs and the such, I discovered MAMP and frankly, I adored it instantly.  After a rather large download, it was effortless to install and running it was a cinch.  And better still, the server applications only run when you open the MAMP application and tell them to run, so that you dont loose vital system resources running services you dont need.

Additionally, my IDE of choice, NetBeans can integrate directly with MAMP, so that when you create (for sake of argument) a new PHP project and run it, it automatically puts the project files into the MAMP Apache directory (to be honest you do need to configure the path when you create the project, but its no big deal) making development painless and convenient.

But I did hit a snag recently when for a university assignment I was required to do a XHTML website using HTML Server Side Includes.  I was shocked when I ran my website and my SSI didnt work.

But there is a solution.

To make .shtml work, I deleted the comment-symbols ( # ) in the file http.conf (find it in /Applications/MAMP/conf/apache/ and at the time of writing was near lines 982:

# To parse .shtml files for server-side includes (SSI):
# (You will also need to add "Includes" to the "Options" directive.)
#
AddType text/html .shtml
AddOutputFilter INCLUDES .shtml