3.2.1 Version control and git

However, we are going to dive in and go straight to learning some of the tools of reproducibility. First, we are going to learn about git. git is version control software. Working with version control basically entails maintaining a single set of files (generally code files, but also other kind of documents, but typically not large binary images, such as rasters) for a project, and making changes to those files as you go in a way that keeps those changes out of sight, but makes it easy to recover them if you need them. For example, the file that we used to create this html document is called unit1-module1.Rmd (it’s an Rmarkdown file, which we will be learning more about shortly) is under version control. I have made many edits to it as I created it, but I have only one version of it visible in the file system. Previous versions of and changes to the file can be found in the prior commits.

The image below illustrates version control–this shows Rstudio’s interface to git, comparing the text in the paragraph above to the text in the previous committed version of this file: the pink shows the older paragraph, the green the newer one (I changed just a few lines in there)

So, that is a small taste of version control. If you have ever worked with Google Docs, it does its own form of version control–you can look back at previous edits and changes. Using git is a bit more explicit and deliberate than that–you have to make commits manually, changes aren’t saved automatically, but that’s okay, and it helps us choose how much we want to change before committing the file version.

So why are we using it here? We want to avoid this common practice used when updating project documents: make a file, name it something, make a bunch of changes, decide you want to keep the previous version -> make a new copy of the file -> append the date to the filename to track the changes. Before long you end up with a cluttered directory that might look something like this:

├── my_folder
    ├── somefile.txt
    ├── Somefile_jan 20 2018.txt
    ├── somefile_2 Feb2018.txt
    ├── somefile_jan12018.txt

You can’t remember what’s in each file or when you made a particular change, and you start to hunt. Version control helps us keep folders neat and pruned, and makes edits fairly recoverable, so that we can just use a single file (in the example above) called somefile.txt and keep track of all the changes made to that file. That’s why we are going to use git in this class.

3.2.2 git and GitHub

git is a program that lives on your local machine. You can make commits locally, and just keep things under version control on your local machine. If you want to collaborate, or even work with your project across multiple machines, you can create a remotely stored version of your git project (repository) on GitHub. GitHub is one of several web services that provides hosting for git repositories. It also provides useful collaboration functionality. We’ll come back to this.

3.2.3 Packages

Packages are another tool of reproducibility that we will learn to use in this class. Packages are bundles of code and documentation that provide a specific functionality for a particular language. Both python and R make extensive use of packages, as the utility of both languages depends heavily on user-contributed code. Code contributions are made through packages, which standardize and formalize the user-developed functions so that they work properly within the language, and can be easily used by the broader community. Packages are also useful even if you don’t intend to contribute code to the community, as they have a particular structure that can be useful for helping to organize and formalize your thinking, data analysis, and reporting. They can also save you time as they make it much easier to access functions that you have developed for one project and apply them to another. I learned these ideas from reading Hillary Parker’s blog post on personal R packages, as well as Karl Broman’s R package primer, and of course Hadley Wickham’s introduction to his R packages book (please read all aforementioned links).

I have adopted the package structure for organizing the analyses and writing of each of my papers. For example, two of my most recent papers are structured as R packages that can be installed from their GitHub repositories (which can be found here and here). These contain all the code used to do the analyses, create the figures, and full (non-journal formatted) versions of the manuscripts. And (as you may have noticed by now), this class is also structured as an R package.

3.3.1 Installations

The first thing you need to do, if you haven’t already done this after visiting the geospaar repository, is to install the materials. The easiest way to do this, because it will give you a full installation of R, Rstudio, and all the necessary packages for working with this material, is to use docker and pull or build the latest image from docker hub. Alternatively, you can get standalone versions of R and Rstudio and use that. We will cover both options here, but prefer to use the docker-based approach.

cd c/My\ Documents/projects/geog246346

git clone https://github.com/agroimpacts/geospaar.git

PORT=8787 # this is the port to run on--you might want to change it
MY_PROJECT_DIRECTORY=c/My\ Documents/projects # change this to yours!!!
./run-container.sh -v $LATEST -p $PORT $MY_PROJECT_DIRECTORY

3.3.2 Your first Rstudio project

At this point, working in either the dockerized Rstudio-server interface, or your local Rstudio, we are going to get started with Rstudio and our reproducibility tools by using some of Rstudio’s handy built-in capabilities. We’ll do that by creating a new Rstudio project that has the minimal set of files and folders needed for an R package, as well as its own git repository.

To do this, with Rstudio: 1. Find the Packages tab, click install, type “devtools” into the “Packages” dialog, and click install. You will notice a bunch of stuff installing in the console (note: skip this step if you are working in the docker image–devtools is already installed) 2. When that is finished, select File > New Project 3. In the new project dialog that pops up, choose “New Directory”, and then in the next page, select the “R package” option at the very bottom. You will see a screen that looks like this:

4. Here we’ll make a couple of decisions:

   • Name the package using your initials (in lowercase) followed by the three digits of the course (246 or 346, depending on which you are enrolled in), e.g. in my case it would be lde346. Do not, one more time, DO NOT, name it using a different convention, e.g. MyFirstNameLastName346. If you do, you will lose points on your assignments.
     
   • Under the “Create project as sub-directory of” enter the folder where you want this project to live. If you are using the docker approach, leave the default ~ in the dialog. Otherwise, if you are working with a local Rstudio install, you should use a folder that is relevant, e.g. one in which you keep all course-related work. I do not suggest your Desktop as a good location, as shown here.
   • Check the option to “Create a git repository”
   • Leave unchecked the “Use renv with this project”
   • Click “Create Project”

5. You will now have an opened Rstudio project, with the name you selected. You are going to be working in this project for the first two units of the class. Let’s configure Rstudio a bit first. Your first view, if you are not using the docker version (assuming this is your first use of Rstudio) will look something like this:

   If this is what you see (because you are not using the docker image, which already provides this view as default) modify the pane layout first, by going to Tools > Global Options > Pane Layout, and make it match this:

   You can also change the Rstudio theme by going to Tools > Global Options > Appearance (I personally prefer light text on dark backgrounds). Also check the Tools > Terminal tab, and make sure that either “Bash” (Mac/Linux) or “Git Bash” (Windows) are enabled in the “New terminals open with” dialog.

   You should also see a terminal tab next to the console tab:

   Last thing under Global Options to modify: > Code > Display > Check Show Margin tab, which should have 80 in the margin column value box.

   If you are using the docker image, feel free to modify the options how you see fit.

6. Now let’s set up some project options that will make our lives easier when building packages (again, this is already done if you are using the docker version). Go to Tools > Project Options, and make it look like this (i.e. check “Generate documentation with Roxygen” and check all boxes in the dialog after click the Configure button):

Okay, we are basically set-up with the full set of tools you need. We are now going to learn to work with git.

4.3.1 SSH keys and access token

First, you will need to create an SSH private-public key pair on your computer, if you don’t have one already, and then add that key to GitHub (you will need to do this for each computer you want to connect to GitHub). The most direct set of instructions, which revolve around RStudio, are found here (and this is generally a great resource covering how to work with git, GitHub, and R/Rstudio) here. To wit, use the following command in the R console:

file.exists("~/.ssh/id_rsa.pub")
file.exists("~/.ssh/id_ed25519.pub")

If the answer to both is “False”, then you will need to create a new SSH key. The easiest way to do that is to go into RStudio Tools > Global Options > Git/SVN and click “Create SSH key”. Leave the passphrase empty, and click through. Keep the ED25519 key type. (See here for more information on ED25519 vs RSA key types)

Press the View public key button above the SSH RSA Key dialog, and copy the text of the key. Then follow steps under “Adding a new SSH key to your account” in this GitHub document to add the SSH key to your GitHub account. You can find your public SSH key by going to Tools -> Global Options -> Git/SVN -> View Public Key.

If you already have an existing SSH key, then I suggest following these instructions here to get it.

Once you have that set up, test your SSH connection following steps 1-4 here, but in place of Step 1 simply go into RStudio’s terminal.

Next, you will also have to set up an access token. To do that, under your GitHub Settings, navigate to the “Developer Settings” on the left hand menu, and then click “Personal Access Token”, and choose “Tokens (classic)”. Fill-out the note with something informative, and choose the expiration date as “custom”, setting it to something just past the end of the semester, and then check the “Repo” box under scopes. Click “Generate Token” at the bottom, and then copy the token and save it somewhere safe (we strongly recommend the use of password management software that keeps passwords in a vault). This token will be the password you have to enter when GitHub prompts you for credentials (see next section). s

4.3.2 Create new private repo on GitHub

For class assignments, you will use a private GitHub repository for your assignments, so that they are only viewable only by you and the course instructors). To set up a private repo, log onto your GitHub account, and, in your Dashboard view (which is where you should be taken), clean the green “New” icon. You will see a dialog that looks like this:

Find the dropdown box on the left that shows your GitHub user name, click on it. In the repository name dialog, enter the same exact name that you used for making your local project repo (i.e. the name of your RStudio project folder, e.g. lde346), select the option to make the repo private, and enter in the Description dialog “GEOG246-346 class repo”. Leave the rest of the fields as they look in this image, then click “Create Repository”. That will bring up another page where you will see your new repository name at the top, followed by two options, one for “Set Up GitHub Copilot” (ignore this), another for Add collaborators to this repository”, followed by three packages of commands. Copy the middle set of commands below the text “…or push an existing repository from the command line”, and then paste them into RStudio’s terminal (note, the lde346.git in the first line will be replaced by the name of your project repo).

git remote add origin https://github.com/ldemaz/lde346.git
git branch -M main
git push -u origin main

Don’t run them yet. First click on the option to add collaborators, and then invite both of us instructors as collaborators to your repo, using our GitHub user names. Invite them to have write access to your repo.

Then run those copied lines from your Rstudio terminal. Note that if you have just set up your SSH keys in RStudio, you might be prompted to enter your GitHub user name and password (the token you generated) after the second command.

Now, go back to the GitHub repository, and refresh the page, and you will see your committed files in there.

4.3.3 Synchronizing with your remote repo

Now that you have a remote repository, you will want to keep it synchronized. The meets that every time you make a commit in your local repo, you should follow it with a push, to make sure the change is committed onto GitHub.

You will notice now, if you look back in RStudio’s git GUI, that the push and pull buttons, which were previously greyed out, are now live. So, that means we can interact with the GitHub repo. Let’s first do a push. Reopen your DESCRIPTION file, and change this line:

Version: 0.0.0.9000

To

Version: 0.0.1

Commit the change with a meaningful message (“Updated package version number”), and then once you have done that, press the “Push” button (which you can access either from the big commit dialog, or in the main git GUI of RStudio). If you go to your repo on GitHub, you will see your new commit message next to your use icon, above the list of files. Have a look at the file itself on GitHub to see that the change is in fact now shown on GitHub.

So that was a “push”. What about a “pull”? A pull is done when a change is made on your GitHub repo that is not on your local repo (i.e. your remote repo is ahead of your local repo). How might that happen? In cases where the remote repo is synced to another local repo on another computer, either because you have connected it to a different computer (e.g. your lab computer), or because another person who has permission to write changes to it has. The act of creating a new local copy of a remote repo is called cloning.

In cases where your local repo is behind your remote repo, you can use the Pull button to bring your remote up to date. That should usually be enough, although conflicts might arise between your local and remote repo, particularly in cases where the remote is much further ahead. We won’t deal with that right now, however.

4.3.4 Cloning the remote repo

Instead, let’s look at how we can use RStudio to create a new project from your remote repo, which you can use to have the project on both your home and lab computer, so you can make changes from either location. This is quite simple.

• On the new machine, which I am assuming already has RStudio set up (in the docker image or standalone), complete with git and ssh keys connected to your GitHub account, you would simply go to File > New Project > Version Control > Git,
• Copy into the “Repository name” dialog at the top the full repo path, which you get by going to the repo’s main page on GitHub and pressing the big green “Code” dialog, and copying the resulting URL string. Note you choose to clone using either HTTPs or SSH, which each give slightly different links. You might have to trial and error to get the one that works, but try SSH to start.
• In the second box (directory name), use the same name as the repo (e.g. lde346), and then choose the directory where you want it live.
• Check open project in new session, and then voila, you have a local version of the repo fully set up.

You now have two local copies of the repo, so when you make a change on one, push it to the remote (on GitHub), and then pull the new changes down to the other.

Another note: you can spoof the process described above by simply cloning the GitHub repo on the computer you used to set it up into a different target directory. If you do that, I recommend that, once you have completed the task and closed out of the relevant RStudio session, that you delete the resulting project folder (to avoid confusion).

4.3.5 Branching

The last thing we are going to do is to set up a new branch in our repo. There is a whole set of instructions how to do that via terminal commands in the branching and merging section of the package help vignette. However, setting up a new branch and syncing it with the remote repo is fairly trivial in newer versions of Rstudio.

• Enter the git GUI
• Press New Branch, add a new branch name, e.g. “test”
• Make sure the “Sync branch with remote” box is checked
• Click “Create”

The new branch will be added locally and to the remote, and you will be changed into the new branch on your local machine.

Similarly, if the new branch was created from one local machine, and there is another local machine that doesn’t yet have it, you can use the same New Branch dialog, but use the Add Remote button to enter the name of remote branch you want and create it locally.

Now you can switch back and forth between branches using the dropdown dialog to the right of the New Branch button (or in terminal, `git checkout ). However, if you make a change to tracked files in the branch you are in and try to switch to another before committing them, you won’t be able to. You have to commit changes first.

To delete the branch, you have to use the terminal (use the RStudio terminal). First switch back into the main branch (using the Rstudio dialog or in terminal running git checkout main), and then run:

git branch -d test
git push origin --delete test

The first command deletes the local branch (named test), the second that branch on the remote repo.

4.3.6 Merging

We will leave the topic of merging for now, but basically merging is quite useful when a small change on one branch (generally a side branch created for the purpose of developing a new feature or fixing a bug) needs to be incorporated into another. An example might be like this:

• Create a new branch off of main branch called “fix/description”, which in our example is to fix several typos in our DESCRIPTION file

• Make the typo fixes to the version of DESCRIPTION in the new fix/description branch, and commit them.
  

• Switch back to the main branch (or git checkout main)

• Merge the changes

  git merge fix/description

• Push the changes to remote, and then pull to other local repos

• If that is the only time you will use the branch, then delete it (locally and to remote)

In reality, that might be too small of a change to make a whole branch for, but it is useful for illustrating the concept.

5.2.1 RMarkdown

We write vignettes using RMarkdown files (note: there is a newer format called Quarto that is now available, but we still use Rmarkdown because Quarto seems to need extra steps for vignettes), which provide a way of mixing text, code, and the output of that code into a variety of document formats, including html, pdf, Word, and various presentation formats. The vignette you are reading now is written in RMarkdown, and you can learn a great deal about how it works by reading the corresponding .Rmd file. But first, have a read of Rstudio’s Overview of Rmarkdown–click “Get Started” and then go through all the linked sections (Introduction through Cheatsheets). This will give a pretty thorough overview of RMarkdown and what it can do. Make sure you pay extra attention to the section on Code Chunks, particularly “Chunk Options”, which control the display and output of the code in your Rmd file.

You can pull up a template for an RMarkdown file by going to File > New File > RMarkdown (a new RStudio install might tell you to update some packages first, which you should do). Name it “Test” in the dialog it brings up, and then save it to your project directory, and then press the “Knit” button at the top of the RStudio window. That will create the html output in a special new window. Delete test.Rmd and test.html when you are finished (we don’t want to track these).

Vignettes require more yaml front matter (that’s the part between the two sets of --- at the top of the RMarkdown file) than the two lines in the template Rmd you just made. These extra lines are used in compiling the Rmd file as a package vignette, and you will also need to run a fairly specific command to build the vignettes along with your package, so that you will find them when you run browseVignettes(). The front matter looks like this:

---
title: "Vignette Title"
author: "Vignette Author"
date: "2024-02-05"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Vignette Title}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

Vignettes also have to be placed within the “vignettes” folder, which you don’t yet have for your project. You could manually make that folder (Files > New Folder), but there is an R package usethis that provides a convenient function (use_vignette, which used to be part of devtools, but has since been moved) for making a vignette folder, template vignette, and some additional modifications needed for vignettes within a single command.

Okay, so let’s run this command

usethis::use_vignette(name = "my_first_vignette")

That should run, but if for some reason the usethis package is not already installed already (it should have been), it won’t work, so Packages > Install > “usethis” (or, in console, install.packages("usthis")) will get it.

The command above will produce the following output:

> usethis::use_vignette(".")
✔ Adding 'knitr' to Suggests field in DESCRIPTION
✔ Setting VignetteBuilder field in DESCRIPTION to 'knitr'
✔ Adding 'rmarkdown' to Suggests field in DESCRIPTION
✔ Creating 'vignettes/'
✔ Adding '*.html', '*.R' to 'vignettes/.gitignore'
✔ Adding 'inst/doc' to '.gitignore'
✔ Creating 'vignettes/my_first_vignette.Rmd'
● Modify 'vignettes/my_first_vignette.Rmd'

That tells us it made three modifications to the DESCRIPTION file (have a look at the file, better yet use the git commit interface to see how DESCRIPTION has changed), created the vignettes folder, added some files to .gitignore, and then created the actual vignette document itself (my_first_vignette.Rmd), which it says we should modify. Open that up and look at it, and you will see it has a bunch of text and code in it. Go ahead and knit it to see what it looks like.

Now modify it. Start by changing the yaml front matter:

---
title: "Overview of the lde346 Package"
author: "Lyndon Estes"
date: "2024-02-05"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Overview}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

Note the places where I have changed the information to make it say something relevant to my package (lde346). Please do the same for your package using its name.

Let’s move down to the text. You will see this code chunk after the yaml:

Keep that, but then replace everything after it to read something like this (make sure to change the package name where needed to match yours, including in the R code chunk call to library():

Go ahead and knit that, and see the result. The chunk should execute and produce the relevant output.

Commit and push the additions/changes, including the new vignette, which was previously untracked. Note that the html file for the vignette does not show up in the git interface, because use_vignette() added any html file in the vignettes folder to the .gitignore file. Vignettes as supposed to build with the package, so no need to track changes to the html since the Rmd is the source file, and more clearly shows the changes (since it is basically a text file, which is most human-readable).

5.2.2 Build your package with the vignette

To get the vignette to knit and install with your package, go into the R console, and run:

devtools::document()
devtools::install(build_vignettes = TRUE)

You should see output that looks very similar to the following in your console:

√  checking for file 'C:\Users\Administrator\Desktop\lde346/DESCRIPTION' ...
-  preparing 'lde346': (438ms)
√  checking DESCRIPTION meta-information ...
-  installing the package to build vignettes
√  creating vignettes (1.9s)
   Warning in as.POSIXlt.POSIXct(x, tz) :
     unable to identify current timezone 'C':
   please set environment variable 'TZ'
-  checking for LF line-endings in source and make files and shell scripts
-  checking for empty or unneeded directories
-  building 'lde346_0.0.1.tar.gz'
   
Running "C:/PROGRA~1/R/R-35~1.2/bin/x64/Rcmd.exe" INSTALL \
  "C:\Users\ADMINI~1\AppData\Local\Temp\2\RtmpYrAZzD/lde346_0.0.1.tar.gz" --install-tests 
|arning in strptime(xx, f, tz = tz) :
 unable to identify current timezone 'C':
please set environment variable 'TZ'
* installing to library 'C:/Program Files/R/R-3.5.2/library'
* installing *source* package 'lde346' ...
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
'lde346'ting help for package     finding HTML links ...
 done
-y_number_checker                       html  
** building package indices
** installing vignettes
** testing if installed package can be loaded
*** arch - i386
*** arch - x64
* DONE (lde346)
In R CMD INSTALL
Reloading attached lde346

And then, as discussed previously using the roxygen2 example using one of the two approaches suggested to find your package vignette, e.g. 

browseVignettes("lde346")  # replace with your package name

Pretty cool, no?

One more way we are going to try this. Checking whether your package can be installed locally. We are going to use another devtools function:

devtools::install_github("agroimpacts/lde346", build_vignettes = TRUE, 
                         auth_token = "paste_your_github_token_here")

One other tweak: in your DESCRIPTION you will see this line:

Depends: R (>= 3.5.2)

The number might be lower than this, reflecting the version of R you were using when creating the package. I recommend it be set to (>= 3.0.0), to allow older versions of R to build the package.

Okay, that’s a wrap for this unit. You just have to do the assignment now.