Data Help with Data GPS/GNSS Data Data Access Search and Access Methods File Server File Server Access Examples Web Services Real-time GPS/GNSS Data Derived Data Products Associated Data Products Data Formats Data Management Data Operations

Related Links

GPS/GNSS File Server Access Examples

The NSF GAGE Facility, operated by EarthScope Consortium, has transitioned from FTP to HTTPS for access to our data archive file system. We are also now requiring user registration and log in for file server access. For more information, see our notices about the transition.

By downloading files from the NSF GAGE Facility, you are agreeing to abide by the NSF GAGE Data Policy.

To browse the HTTPS file server, simply visit GPS/GNSS file server to log in (see User Authentication below) and view our holdings.

Please send your issues, comments, and suggestions to: data-help@earthscope.org.

User Authentication

We now require all data users to register and login to access the GAGE Data File Server (this new authentication system is known as Identity Management or simply IdM). Please see the link above to the notice. To access the file server from the command line you will need to:

  1. Register a user account with us, if you haven't already, to create a profile.
  2. Access data:

    • For casual access

      • Use a web browser to login to your account and browse data files and download them.

        OR

      • Include an authorization header parameter with your command line data file requests that includes your access token:
        • Use a http file transfer client (Wget or cURL) with your temporary access key. From your profile API Key page, you can view and copy your short-lived access token for trial and testing purposes. You can copy and paste this token into a http request header: 
              --header "Authorization: Bearer <PASTE YOUR ACCESS TOKEN HERE>"

    • For scripted access

      Good for scripts that run regularly and unattended to access data files.
      1. Command Line Interface (CLI)
        1. On standard servers:
          1. Download and setup our Python based Command Line Interface (CLI) on your server to simplify access token acquisition, maintenance, and usage (Requires Python 3.7+ and a bash-like shell). See the README for usage instructions
          2. Use the CLI to log in and, via command substitution, automatically insert valid (refreshed) tokens in your request's header parameter:
            --header "Authorization: Bearer $(es sso access --token)"

            NOTE: your shell environment may require different command substitution syntax, e.g., using backticks, e.g.,
            --header "Authorization: Bearer `es sso access --token`".

            TROUBLESHOOTING
            • If you are having trouble with command substitution, try running the es sso access --token command by itself to see if the token is being returned correctly. If it is, then the issue is likely with the command substitution syntax in your shell environment. See above note.
            • If you get an error regarding not being able to refresh your tokens, run the following EarthScope CLI commands and get new tokens:
              1. es sso logout
              2. es sso login
        2. For containers without persistent storage:
          1. You basically will want to share your access tokens across your servers. You will keep a centralized server or lambda function's CLI tokens up-to-date daily via cron and make those tokens available to your other servers/containers via persistent storage or a secrets manager.
      2. Software Development Kit (SDK)
        • For python script developers who want to authenticate their python scripts without the CLI.
        • See our SDK code repository README which includes an example python download script utilizing our SDK.
        • Note: For non-Python coders, you can look at the SDK's Python code as an example of the basic logic needed to implement similar functionality in your coding language of choice. You can also see Auth0's quickstarts.
  3. See the access examples below that utilize our CLI with both cURL and Wget. See our SDK code repository README for an example python download script.

Examples

The following examples illustrate access to the NSF GAGE Facility GPS/GNSS data file server using standard file/data transfer utilities in a bash-like shell (see bash for Windows). See our SDK code repository README for an example python download script.

Client HTTPS Access command syntax

cURL

 Get a single file
    Options:
  • -L (follow redirects)
  • -O (uses server filename)
  • -f ((HTTP) Fail on error without server output. Error code 22 -- good for scripting)
    •
Include an authorization header parameter with your bearer token. Install our Command Line Interface to facilitate acquiring and refreshing your access token. You can then simply use the CLI command "es sso access --token" via command substitution, for a bash example:
$ curl -L -O -f https://gage-data.earthscope.org/archive/gnss/rinex/obs/2022/060/p1230600.22d.Z --header "Authorization: Bearer $(es sso access --token)"
Get multiple files from a list
(build your list of files using: GNSS Data Access Notebooks or query parameters noted below)		 
$ xargs -n 1 curl -O --header "Authorization: Bearer $(es sso access --token)" < file_urls_to_download.txt
Get a directory listing
(using query parameters noted below) 
$ curl -L "https://gage-data.earthscope.org/archive/gnss/rinex/obs/2022/060/?list" --header "Authorization: Bearer $(es sso access --token)"

Download matching files

Downloads three files ('p1230600.22d.Z', 'p1240600.22d.Z', 'p1260600.22d.Z') from archive directory /archive/gnss/rinex/obs/2022/060/ .

 
$ curl -L -O "https://gage-data.earthscope.org/archive/gnss/rinex/obs/2022/060/p12{3,4,6}0600.22d.Z" --header "Authorization: Bearer $(es sso access --token)"

Wget

 Get a directory of data Include an authorization header parameter with your bearer token. Install our Command Line Interface to facilitate acquiring and refreshing your access token. You can then simply use the CLI command "es sso access --token" via command substitution:
$ wget -N -r -np -erobots=off --reject="tmp,index.html*" https://gage-data.earthscope.org/archive/gnss/rinex/obs/2022/060/ --header "Authorization: Bearer $(es sso access --token)"
Get a single file 
$ wget https://gage-data.earthscope.org/archive/gnss/rinex/obs/2022/060/p1230600.22d.Z --header "Authorization: Bearer $(es sso access --token)"
Get multiple files from a list
(build your list of files using: GNSS Data Access Notebooks or query parameters noted below)		 
$ wget -i file_urls_to_download.txt --header "Authorization: Bearer $(es sso access --token)"
Get a directory listing
(using query parameters noted below) 
$ wget "https://gage-data.earthscope.org/archive/gnss/rinex/obs/2022/060/?list&dirs" --header "Authorization: Bearer $(es sso access --token)"
Get station P123 obs files for the year 2010
    Options:
  • --accept, -A (match files w/suffixes or patterns)
  • --level (number of subdirectories to recurse into)
  • -nd (Don't create hierarchy of dirs when retrieving recursively)
    •  
$ wget -N -r -l3 -np -nd -erobots=off --reject="tmp,index.html*" -A "p123*" https://gage-data.earthscope.org/archive/gnss/rinex/obs/2020/060/ --header "Authorization: Bearer $(es sso access --token)"

Query parameters


NOTE: this feature is a Beta release. Let us know of issues or features you would like to see here.

 
?list

Append ?list to directory URLs to get plaintext directory listings w/file names, file sizes in bytes, ISO8601 modification dates, and Unix timestamps. Use this parameter with cURL or Wget to check for new files in our archive. This does not display sub-directories.

NOTE: When using cURL or wget, etc., be sure to put your full request URI including query parameters in quotes, or individually escape the special characters \? and \&, to avoid confusing your shell.

 
$ wget "https://gage-data.earthscope.org/archive/gnss/rinex/obs/2022/060/?list" --header "Authorization: Bearer $(es sso access --token)"
See https://gage-data.earthscope.org/archive/gnss/rinex/met/2021/072/?list.
Note: must be logged in first, otherwise the browser log in process strips these parameters from the URL.

Add additional parameter to the list option for a listing with sub-directories shown: 

&dirs

 
$ wget "https://gage-data.earthscope.org/archive/gnss/products/position/?list&dirs" --header "Authorization: Bearer $(es sso access --token)"
See https://gage-data.earthscope.org/archive/gnss/products/position/?list&dirs.
Note: must be logged in first, otherwise the browser log in process strips these parameters from the URL.

Add additional parameter to the list option for a listing of full URL file paths: 

&uris

This can be a good way to build a list of URLs to feed to Wget or cURL (see also GNSS Data Access Notebooks for generating lists of files based on search parameters like time, location, etc).

 
$ wget "https://gage-data.earthscope.org/archive/gnss/rinex/obs/2022/060/?list&uris" --header "Authorization: Bearer $(es sso access --token)"
See https://gage-data.earthscope.org/archive/gnss/rinex/obs/2022/060/?list&uris.
Note: must be logged in first, otherwise the browser log in process strips these parameters from the URL.

Combine parameters with the list option for a listing of URLs with sub-directories shown: 

&dirs&uris

 
$ wget "https://gage-data.earthscope.org/archive/gnss/products/position/?list&dirs&uris" --header "Authorization: Bearer $(es sso access --token)"
See https://gage-data.earthscope.org/archive/gnss/products/position/?list&dirs&uris.
Note: must be logged in first, otherwise the browser log in process strips these parameters from the URL.
 

Last modified: 2025-01-17  17:07:53  America/Denver