Connecting to CloudSql with App Engine and Golang

Things that were not immediately obvious.

What wasn't immediately clear was that app engine was not able to connect to the 'new' Managed VM version of Cloud Sql.

In order to use app engine and connect to cloud sql you need to import the following 2 packages as side effects, as well as the "database/sql" package:

import (
 "database/sql"
 _ "appengine/cloudsql"
 _ "github.com/go-sql-driver/mysql"
)

NOTE: please use access control; if you set the root password for the localhost domain, your app engine instance will not be able to connect with standard cloud sql DSN, such as:

user@cloudsql(project-id:instance-name)/dbname

in the case of the "go-sql-driver/mysql" package.

Upload / Download to Google Cloud Storage with the Python API

It took a bit of experimentation but here's a way to upload large files to the Google Storage API:

	import io
	import logging
	logging.getLogger('googleapiclient.discovery').setLevel(logging.WARNING)
	logging.getLogger('oauth2client.client').setLevel(logging.WARNING)
	from apiclient.discovery import build
	from apiclient.http import (
	MediaIoBaseDownload,
	MediaFileUpload,
	)
	from oauth2client.client import GoogleCredentials
	LOGGER = logging.getLogger('storage')
	def get_storage():
	credentials = GoogleCredentials.get_application_default()
	return build('storage', 'v1', credentials=credentials)
	def download(bucket, object_name, to_path):
	service = get_storage()
	request = service.objects().get_media(bucket=bucket,
	object=object_name)
	fh = io.FileIO(to_path, mode='w')
	downloader = MediaIoBaseDownload(fh, request, chunksize=1024 * 1024)
	done = False
	while done is False:
	status, done = downloader.next_chunk()
	if status:
	LOGGER.info("Download %d%%." % int(status.progress() * 100))
	LOGGER.info("Download Complete!")
	def upload(file_path, fn, bucket):
	media = MediaFileUpload(file_path, mimetype='application/octet-stream',
	chunksize=1024 * 1024, resumable=True)
	service = get_storage()
	o = service.objects()
	o.insert(name=fn, bucket=bucket, media_body=media).execute()

view raw storage.py hosted with ❤ by GitHub

emacs mode inside ipython(jupyter) notebook

Here's a github gist that you'll need to paste into

custom.js

which is located in

~/.ipython/profile_default/static/custom/custom.js

	// register a callback when the IPython.notebook instance is created.
	$([IPython.events]).on('app_initialized.NotebookApp', function(){
	function to(mode) {
	// this can be either 'vim' or 'emacs'
	var mode = mode || 'emacs';
	// first let's apply mode to all current cells
	function to_mode(c) { return c.code_mirror.setOption('keyMap', mode);};
	var cells = IPython.notebook.get_cells();
	if (cells != null) {
	cells.map(to_mode);
	}
	// apply the mode to future cells created
	IPython.Cell.options_default.cm_config.keyMap = mode;
	};
	require(["codemirror/keymap/emacs"],
	function (_) {
	if (IPython.notebook != undefined) {
	to('emacs');
	};
	});
	});

view raw configure_emacs.js hosted with ❤ by GitHub

This works for 4.0.0.dev :)

python pid file

Install lockfile:
$ pip install lockfile
Choose your path, remember is has to be writable by the user who invokes your python program. Here's an example:

	import sys
	from time import sleep
	from lockfile.pidlockfile import PIDLockFile
	if __name__ == '__main__':
	# If we can't acquire a lock in 1 second, raise an exception.
	with PIDLockFile('/home/me/script.pid', timeout=1):
	while True:
	sys.stdout.write('Check /home/me/script.pid for my process id\n')
	sleep(1)

view raw lockfile.py hosted with ❤ by GitHub

As long as the process is running there will be a file named /home/me/script.pid containing the process id; this PID file can be used as a lock file to prevent multiple processes from running at the same time, in case that wasn't already obvious :)
https://github.com/openstack/pylockfile

SSH tunneling for SparkUI

When starting a Spark session you can visit the status page on port 4040 by default. For instance when you start the pyspark program you'll see:

14/10/06 05:22:03 INFO Utils: Successfully started service 'SparkUI' on port 4040.

I don't want to allow public access to this page so I'll use SSH to forward the port. I have port 8125 open for tcp connections on my gcloud instance. First, on the gcloud instance, forward the Spark service's port 4040 to port 8125, which is open to the Internet; while logged in to the gcloud instance run:

 ssh -4 -f localhost -L 8125:localhost:4040 -N

Next, on your laptop you'll want to forward the gcloud Internet available opened port at 8125 to a port like 2200 so that we can visit the page locally. On your laptop run:

ssh -N -L 2200:localhost:8125 10.10.10.10

replace '10.10.10.10' with the Internet facing IP of your gcloud instance.

Now visit http://localhost:2200 and the service that is running on your gcloud instance on port 4040 is first forwarded to 8125 on that host, and then forwarded to port 2200 locally. 

References:

http://www.augustcouncil.com/~tgibson/tutorial/tunneling_tutorial.html

Method Covariance in Scala

Consider these base classes

	class Dirt
	class GardenItem extends Dirt {
	val flavor = "seedy"
	}
	class Cucumber extends GardenItem {
	override val flavor = "bland"
	}
	class Watermelon extends GardenItem {
	override val flavor = "sweet"
	}
	class Cocktail extends Cucumber {
	override val flavor = "refreshing"
	}
	class FruitSalad extends Watermelon {
	override val flavor = "fruity"
	}

view raw Base.scala hosted with ❤ by GitHub

Let's look at this method that has a covariant return value for the class GardenItem:

	/**
	Here is an object that has a method mixFoods which will accept
	a subclass of a GardenItem and return a list of either
	GardenItems or subclasses of GardenItems.
	*/
	object CovarianceExample {
	/**
	Covariance example (compiles)
	def mixFoods[T >: GardenItem](food: T): List[T] = {
	This method definition states that within the context of the
	mixFoods method, there is a type T that is a superclass of a GardenItem.
	We then go on to say that this method accepts an argument which
	is of type T, a subclass of GardenItem and it returns a List of
	either GardenItems or subclasses of GardenItems.
	*/
	def mixFoods[T >: GardenItem](food: T): List[T] = {
	food match {
	case x: FruitSalad => {
	/**
	Watermelon is a subclass of GardenItem so this is valid
	to the compiler.
	*/
	List(new Watermelon, food)
	}
	case x: Cucumber => {
	/**
	Cocktail is a subclass of Cucumber, which is a subclass of
	GardenItem so this is also valid to the compiler.
	*/
	List(food, new Cocktail)
	}
	case _ => List()
	}
	}
	}

view raw Covar.scala hosted with ❤ by GitHub

And an example that will not compile:

	/**
	Here's an example of failing to return the correct type
	*/
	object CovarianceExampleFailFirst {
	/**
	Covariance example (does not compile)
	def mixFoods[T >: Watermelon](food: T): List[T] = {
	This method definition states that within the context of
	the mixFoodsFail method, there is a type T that is either
	a Watermelon or a subclass of Watermelon.
	Given what we know about type T, this method will accept an argument
	of type T and return a list of type T.
	Unfortuanately, this code will not compile since we are
	attempting to return a GardenItem within the list of what
	can only contain type T.
	*/
	def mixFoods[T >: Watermelon](food: T): List[T] = {
	food match {
	case x: FruitSalad => {
	/**
	This will not compile because GardenItem is not a
	subclass of Watermelon; it's a superclass of Watermelon.
	*/
	List(new GardenItem, food) // Fails!
	}
	case _ => List()
	}
	}
	}

view raw CovarFail.scala hosted with ❤ by GitHub

references:
- Type Basics
- Covariance and Contravariance in Scala/

Multiple Java versions on my mac

I have Java 8 JDK installed and I would like to use Scala but it works much better with Java 7 JDK at the moment. My solution was to install Java 7 SDK and create bash functions in my bash startup file, in my case .bashrc

function java8() { 
   export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home } 

function java7() { 
   export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.7.0_67.jdk/Contents/Home }

The above functions swap the JAVA_HOME environment variable which causes the respective Java version to be used[1]. Example:


computer ~ $ java8 
computer ~ $ java -version 
java version "1.8.0_05" 
Java(TM) SE Runtime Environment (build 1.8.0_05-b13) 
Java HotSpot(TM) 64-Bit Server VM (build 25.5-b02, mixed mode) 
computer ~ $ java7
computer ~ $ java -version 
java version "1.7.0_67" 
Java(TM) SE Runtime Environment (build 1.7.0_67-b01) 
Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)


[1] http://java.dzone.com/articles/multiple-versions-java-os-x