Connecting to CloudSql with App Engine and Golang

Things that were not immediately obvious.

What wasn't immediately clear was that app engine was not able to connect to the 'new' Managed VM version of Cloud Sql.

In order to use app engine and connect to cloud sql you need to import the following 2 packages as side effects, as well as the "database/sql" package:

import (
 "database/sql"
 _ "appengine/cloudsql"
 _ "github.com/go-sql-driver/mysql"
)

NOTE: please use access control; if you set the root password for the localhost domain, your app engine instance will not be able to connect with standard cloud sql DSN, such as:

user@cloudsql(project-id:instance-name)/dbname

in the case of the "go-sql-driver/mysql" package.

Upload / Download to Google Cloud Storage with the Python API

It took a bit of experimentation but here's a way to upload large files to the Google Storage API:

emacs mode inside ipython(jupyter) notebook

Here's a github gist that you'll need to paste into

custom.js

which is located in

~/.ipython/profile_default/static/custom/custom.js

This works for 4.0.0.dev :)

python pid file

Install lockfile:
$ pip install lockfile
Choose your path, remember is has to be writable by the user who invokes your python program. Here's an example:
As long as the process is running there will be a file named /home/me/script.pid containing the process id; this PID file can be used as a lock file to prevent multiple processes from running at the same time, in case that wasn't already obvious :)
https://github.com/openstack/pylockfile

SSH tunneling for SparkUI

When starting a Spark session you can visit the status page on port 4040 by default. For instance when you start the pyspark program you'll see:

14/10/06 05:22:03 INFO Utils: Successfully started service 'SparkUI' on port 4040.

I don't want to allow public access to this page so I'll use SSH to forward the port. I have port 8125 open for tcp connections on my gcloud instance. First, on the gcloud instance, forward the Spark service's port 4040 to port 8125, which is open to the Internet; while logged in to the gcloud instance run:

 ssh -4 -f localhost -L 8125:localhost:4040 -N

Next, on your laptop you'll want to forward the gcloud Internet available opened port at 8125 to a port like 2200 so that we can visit the page locally. On your laptop run:

ssh -N -L 2200:localhost:8125 10.10.10.10

replace '10.10.10.10' with the Internet facing IP of your gcloud instance.

Now visit http://localhost:2200 and the service that is running on your gcloud instance on port 4040 is first forwarded to 8125 on that host, and then forwarded to port 2200 locally. 

References:

http://www.augustcouncil.com/~tgibson/tutorial/tunneling_tutorial.html

Method Covariance in Scala

Consider these base classes

Let's look at this method that has a covariant return value for the class GardenItem:

And an example that will not compile:



references:
- Type Basics
- Covariance and Contravariance in Scala/

Multiple Java versions on my mac

I have Java 8 JDK installed and I would like to use Scala but it works much better with Java 7 JDK at the moment. My solution was to install Java 7 SDK and create bash functions in my bash startup file, in my case .bashrc

function java8() { 
   export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home } 

function java7() { 
   export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.7.0_67.jdk/Contents/Home }

The above functions swap the JAVA_HOME environment variable which causes the respective Java version to be used[1]. Example:


computer ~ $ java8 
computer ~ $ java -version 
java version "1.8.0_05" 
Java(TM) SE Runtime Environment (build 1.8.0_05-b13) 
Java HotSpot(TM) 64-Bit Server VM (build 25.5-b02, mixed mode) 
computer ~ $ java7
computer ~ $ java -version 
java version "1.7.0_67" 
Java(TM) SE Runtime Environment (build 1.7.0_67-b01) 
Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)


[1] http://java.dzone.com/articles/multiple-versions-java-os-x