Source: EarthEngine Data Catalogue

Getting Git Right on Google Earth Engine

So this came as the need to simply being able to replicate Google Earth Engine repositories, scripts, and codes. Turns out you could always do that, go to https://earthengine.googlesource.com/ and find your repo and then download it using git clone or downloading the tar zipped file.

Of course, you can only download the repos to which you have either Owner, Reader or Writer permission. This went back to my earlier project on geetakeout and the story trying to move from my university account to my personal account. While the tool established a way for you to do this, I feel a revisit was much needed for two reasons.

• First of all, the method copied only Owner repos, for the reader and writer accounts to be transferred the owners of those accounts would have to readd you as readers and writers. Also we were only replicating owner repos initially.
• The second and somewhat annoying bug with Git on windows, if you have a folder inside a repo, git clone will fail because of the way in which paths are parsed. This is not an issue on Linux on mac as far as I know. I am sure someone has a quick fix, but I could not find one out quickly enough :) . Here is an example

Git Clone issue with Windows

Hence the tinkering with what could be done and I thought what if we could create a tool that gets a snapshot of all of your repos including the reader, writer, and owner repos and scripts and works on Windows as well as other OS. However please note that this tool depends on Selenium and may have issues with some user accounts, with two factor enabled, or where less secured apps are not allowed.

Repo Snapshot

I decided to create this as a standalone tool for now and will integrate this to gee asset manager & addon and geetakeout tools in the future as needed. The tool is simply designed to parse through and first figure out your repos and the permissions that come along with it.

Turns out the code.earthengine.google.com page contains this information, and this needed to be parsed out. Since you are creating snapshots of your account repos, the date of snapshots is important and this tool creates three repositories within the destination folder.

owner_data, reader_date, and writer_date

Subfolders created from the process

File Handling and Cleanup

The files are downloaded as a tar.gz file and it turns out unzipping in python runs some issues if the folder paths are nonstandard. However, the tools automatically unzip the folder and delete the tar.gz files to give you clean repo snapshots.

Extracted Repos with Files

Since these are zip extracts instead of Git Clone operation it bypasses the git clone failures for windows and it clones all repo types with permissions. There you go, this is a small tool but useful if you would like to keep a local copy of your scripts.

Wrapping it up: Standalones

Now I know what you are thinking you can probably push all of these from one git to your personal Github account just add a different remote repo and push for example as discussed here. I have not tested this out so I cannot vouch if this works and also for now even if private repos are free now on GitHub I might still want a local copy on my machine.

The final tool looks like this in action, you can decide to run the browser in a non-headless manner meaning you can get firefox to login and run these operations as you watch things download. You can download this and hopefully more tools in my earthengine-standalone-tools repo. There are two setup steps and it assumes you are using Python 3.

It takes a few minutes so be patient

• install the requirements.txt file by migrating to the main earthengine-standalone-tools folder and using

pip install -r requirements.text

• download the gecko drivers for selenium by now migrating into the eereposnap folder and simply using

python eereposnap.py 

Getting the tool setup and running it

Finally, run using

python eereposnap.py "path to folder" 
or
python eereposnap.py "path to folder" "active"

eereposnap download files and folders

You can find the standalone tool folder and tool here. Google Earth Engine is forgiving mostly, you can mostly undo what you have done and tinkering is useful. I hope some of you may find this tool useful and feel free to keep an eye out for more standalone applications to be added to the repo. Star the repo so I can keep working on some other fun projects and add-ons.