File Transfer: Globus
One can use these tools to transfer files between and within machines/computers; here, our focus is ARC clusters.
How to use Globus for copying (very large) files between computers (e.g., onto and off of ARC clusters) with fault tolerance and a UI.
Synopsis: Globus is a very powerful tool. While the other file transfer tools can complete their work with a single command, Globus is more of a file transfer environment. Hence it is more complicated than the other tools and has higher start-up cost for you. But do not be put off: it is not that complicated and it is VERY powerful. There is a reason why Globus is the defacto standard tool for large file transfers. Here we present a sequence of videos, which should be watched in order, because some videos depend on content from previous videos in this series. We have broken the information into shorter videos so that users can “enter” this sequence where it is appropriate for them.
Globus overview. For those who do not know Globus or want a refresher in what it does.
Globus prerequisites. Complete these steps (which are required regardless of Globus use) before working with Globus—it will make your life much easier. NEED TO ADD HOW TO MAKE PROJECT AREA VISIBLE TO GLOBUS USING COLDFRONT.
Roadmap of Globus-specific videos. This short video demonstrates the unified view in presenting primary globus features in the remaining videos.
Unlinking previous Globus identity. This video is only relevant for those people coming to VT from another institution where they had a Globus account. Because in general your Globus identity is intimately tied to your present institution, and you had a Globus account at a previous institution, you must update your Globus account so that it is affiliated with your new institution (which is presumably Virginia Tech). This video shows how to “unlink” from your previous university. Then, in videos below, you can follow those instructions to establish your new VT-affiliated Globus identity, which are the same steps as creating a VT-affiliated Globus account for someone who has never had a Globus account.
Create a Globus account and log in. This is used by new Globus users and those Globus users who have come from another institution and have unlinked their previous Globus identity. We also ensure that you can see the main VT collection.
File transfer demonstration using Globus. This demo is a file transfer within one collection on one cluster. The explanation is generalized to illustrate how to perform file transfers between two clusters using different collections. This demo also shows how the directories Globus displays are the same as those under “/projects” on the ARC clusters.
Install Globus Connect Personal (GCP) on your laptop. GCP is used with Globus to enable file transfers between your local computer and ARC clusters.
File transfer demonstration using Globus and Globus Connect Personal (GCP). GCP is used with Globus to enable file transfers between your local computer and ARC clusters.
Setup Globus Guest Collections. Globus Guest Collections (GGCs) are areas of storage that are accessible to you and other people that you designate. These “others” can be colleagues at other institutions around the world, making for powerful file sharing.
Setting permissions on files within Globus collections. Just because some directory within a collection is visible to Globus does NOT mean that all of the files and directories within the collection are visible. You can still use Linux/Unix permissions to control Globus’ access to your files and directories. This video shows the issues and how you can control file and directory visibility.