Google Colab Export

Colab Swift GDrive Service Account

Background FAST.AI provided a wonderful course in 2019 in two parts. The second part introduced Swift. Swift is a standard 3G language with lots of safety checks to stop me making all the mistakes I normally make when learning a new language. Unfortunately “How do I do?” involves Google and guess.

Colab is a free resource of server with GPU from Google but you only have a fresh local drive.

Google drive (Gdrive) is a 15GB free resource where you can store data.

FAST.AI generally uses Python. Python running on Colab allows you to connect to your Google Drive. GDrive Colab and Python - a great combination. Python invokes Gdrive authentication which invokes a data entry box into which you sign and returns a token. You paste the token into Colab and everything is fine.

So all we need to do is write similar functionality for Swift and we have a great combination but with the power of Swift.

So I will write a program and ask you to type in your Google userid and password and then I can ask Google to give me complete access to your Gdrive.

I can not seeing that being too popular.

Fear not Google to the rescue. You can have a service acccount as well as your normal Google userid. Service accounts have a e-mail like userid and a private key rather than a password. You can pick a Google folder on your Gdrive and allow access to the service account using the email like userid.

You can disable your individual service account from your collection.

To use the service account to access the GDrive you build a request called a JWT which is signed with the private key and has a maximum life of 60 minutes.

The service account administration is found at https://console.developers.google.com

So we can access the Gdrive but how?

Enter the Google API. This is a series of GET POST PUT request using HTTP which you can issue from Swift URLSession request.

Swift is great for mobile phones because it just waits for input from the screen or an event.

A program running on Colab has to wait for the Swift URLSession to complete, examine the HTTP status information and process any response.

I hope this is the correct method of referral. A gentleman called pcuenca has written a export import function for Swift Colab similar to Python.

This allows the Jupyter notebook to be marked with “// export” in individual cells and pcuenca (https://github.com/latenitesoft/NotebookExport) generates a Package.swift and main.swift which can be imported into the next notebook using

%install ‘.package(path: “/content/FastaiNotebook_ExportTest”)’ FastaiNotebook_ExportTest

The problem with Colab is the server is empty and /content contains nothing.

So here is an outline of the solution

// export

Save the notebook on your GDrive Copy the notebook to /content

Allow pcuena to generate the Package and main on /content by reading the notebook on /content. It is a JSON file. Copy the Package and main to your Gdrive

Start a new notebook. Copy the files back to /content Restart the notebook Add to the first cell

%install ‘.package(path: “/content/FastaiNotebook_ExportTest”)’ FastaiNotebook_ExportTest import FastaiNotebook_ExportTest

%install recursive follows the imports so it is necesary to copy all the exported data from GDrive

Now for the detail.

So first visit https://console.developers.google.com Create a project and make a service account. This creates a JSON file containing a number of things including the private key and client email. Save this on your local machine or GDrive. Go to your GDrive and find the folder Colab Notebooks and allow access to the folder for the client email and untick the Notify People otherwise you get an email bounce.

Now build a swift variable

let myjson=[ “private_key”: “—–BEGIN PRIVATE KEY- “client_email”: “gdswift@swift3-257813 ] as [String:String]

Create a new notebook

%install ‘.package(url: “https://github.com/mxcl/Path.swift”, from: “0.16.1”)’ Path

%install ‘.package(url: “https://github.com/IBM-Swift/Swift-JWT.git”, from: “3.0.0”)’ SwiftJWT

%install ‘.package(url: “https://github.com/saeta/Just”, from: “0.7.2”)’ Just

%install ‘.package(url: “https://github.com/latenitesoft/NotebookExport”, from: “0.5.0”)’ NotebookExport

%install ‘.package(url: “https://github.com/Conwyn/GCE”, from: “0.0.0”)’ GCE

import Foundation import FoundationNetworking import Path import Just import SwiftJWT import NotebookExport import GCE

let myjson=[ “private_key”: “—–BEGIN PRIVATE KEY-

“client_email”: “gdswift@swift3-257813 ] as [String:String]

struct MyClaims: Claims { var iss: String var aud: String var exp: Date
var iat: Date var scope: String}

public extension String { @discardableResult func shell(_ args: String…) -> String { let (task,pipe) = (Process(),Pipe()) task.executableURL = URL(fileURLWithPath: self) (task.arguments,task.standardOutput) = (args,pipe) do { try task.run() } catch { print(“Unexpected error: (error).”) } let data = pipe.fileHandleForReading.readDataToEndOfFile() return String(data: data, encoding: String.Encoding.utf8) ?? “” } }

Copy Gdrive to /content

GCEImport(myjson: myjson , importName: “FastaiNotebook_NBE1” )

GCEImport(myjson: myjson , importName: “FastaiNotebook_NBE2” )

GCEImport(myjson: myjson , importName: “FastaiNotebook_NBE3” )

Then restart the notebook and add to the first cell %install ‘.package(path: “/content/FastaiNotebook_ExportX”)’ FastaiNotebook_ExportX where X is the last export referring to previous. So 3 imports 2 which imports 1. The %import only requires one statement the GCEImport are required for each file.

import FastaiNotebook_NBE1

import FastaiNotebook_NBE2

import FastaiNotebook_NBE3

// export import Path public func dummytest3(NBn:String)->String{ return NBn }

Find the existing notebook name NBn and copy it to /content

let NBn = GCEPreExport(myjson:myjson)

let exporter = NotebookExport(Path.cwd/NBn)

print(exporter.export(usingPrefix: “FastaiNotebook_”))

print(“/bin/ls”.shell(“-lh”))

Copy back to GDrive

GCEPostExport(myjson:myjson)

#Acknowledgement

To korakot for obtaining the notebook name on Colab

from requests import get filename = get(‘http://172.28.0.2:9000/api/sessions’).json()[0][‘name’]