Real World APIs: Snagging a Global Entry Interview

As my new job will have me traveling a bit more often, I finally bit the bullet and signed up for Global Entry (which is similar to TSA PreCheck but works for international travel as well). A few days after submitting my application and payment, I was conditionally approved. The next step was to schedule an “interview,” which is essentially a 10-minute appointment where they ask a few questions and take biometrics. The interview must be done in person at one of relatively few CBP locations.

Here in Raleigh, North Carolina, my two closest locations are Richmond and Charlotte. Unfortunately, CBP’s scheduling portal indicated no availability for new appointments at either location. No additional context is provided, so I have no idea whether I should keep trying every few days, or attempt to schedule an appointment at a remote location to coincide with future travel.

My only hope at this point is that spots will eventually open up as other applicants cancel their appointments or CBP adds sufficient staff to meet demand. But that means manually logging into the portal, completing two-factor authentication, and checking both of my desired appointment locations each and every time.

Sounds like a great use case for automation, doesn’t it?

Discovering an API

Being curious as I am, I hit F12 to open Firefox’s developer console, selected the Network tab, and clicked on one of the appointment locations again. This captured the following request:

GET https://ttp.cbp.dhs.gov/schedulerapi/slots?orderBy=soonest&limit=1&locationId=14321&minimum=1

The response was [] — the JSON representation of an empty list. Interesting. Let’s try a site that does have an available appointment: Newark, NJ. (Notice the different site ID specified in the query parameters.)

GET https://ttp.cbp.dhs.gov/schedulerapi/slots?orderBy=soonest&limit=1&locationId=5444&minimum=1

This returned the following:

[ {
  "locationId" : 5444,
  "startTimestamp" : "2019-08-19T14:00",
  "endTimestamp" : "2019-08-19T14:15",
  "active" : true,
  "duration" : 15
} ]

Oh, that’s much more interesting. Here we see a list containing a single object: The representation of an available 15-minute appointment on August 19th at 2:00 PM. This looks very promising. Next step: Which of the original request headers are necessary to recreate the request? (I’m authenticated to the portal, so the request includes my session authentication as it is sent by the web browser.) Let’s try a bare request using CURL (note the quotes wrapping the URL):

$ curl "https://ttp.cbp.dhs.gov/schedulerapi/slots?orderBy=soonest&limit=1&locationId=5444&minimum=1"
[ {
  "locationId" : 5444,
  "startTimestamp" : "2019-08-19T14:45",
  "endTimestamp" : "2019-08-19T15:00",
  "active" : true,
  "duration" : 15
} ]

Well that was easy! It turns out that authentication is not necessary to call the API. This will greatly simplify the work needed to automate requests.

Most of the fields contained in the appointment representation are self-explanatory: timestamps for the start and end time, a boolean for whether the slot is active, and a duration (which the timestamps indicate is measured in minutes). But what about locationId? The number 5444 isn’t very helpful. Can we use the API to map numeric location IDs to human-friendly names?

Normally, this is where we’d turn to the documentation, but unfortunately this API doesn’t seem to have any public documentation as it likely was not intended to be consumed outside of the website. So, let’s start by guessing where an endpoint for listing locations might live. Stripping away the query parameters, our current endpoint is

https://ttp.cbp.dhs.gov/schedulerapi/slots

Some URLs that I tried were

https://ttp.cbp.dhs.gov/schedulerapi/locations - 403
https://ttp.cbp.dhs.gov/schedulerapi/locationids - 403
https://ttp.cbp.dhs.gov/schedulerapi/sites - 403
https://ttp.cbp.dhs.gov/schedulerapi - Redirects to main website

I wasn’t getting anywhere by guessing, so I decided to just google the root API URL. After scrolling past the various official CBP webpages, I found a GitHub project called next_global_entry. Not surprisingly, I was not the first person to discover the TTP API.

Scheduling an interview for your Global Entry application is hard. Some enrollment centers are months out. But they tend to sneakily add slots in the near future. This script will alert you when that happens.

Very cool! The author doesn’t mention how he found it, but the project’s README notes a new endpoint: https://ttp.cbp.dhs.gov/schedulerapi/slots/asLocations Opening the endpoint in Firefox, we receive a JSON representation of ten individual sites complete with name, location, timezone, and other details.

The sites are indexed 0 through 9, but there are clearly more than ten appointment sites. It seems the API is paginating its results; displaying only the first ten sites by default. How do we get it to return more?

Looking back at the first request we captured, we notice that one of the query parameters was limit=1. This parameter is instructing the API to return only a certain number of objects (in this case, a single object). This is a very common implementation, usually coupled with an offset parameter to enable clients to navigate through a long list of results one “page” at a time. So let’s try specifying a limit on our new endpoint:

https://ttp.cbp.dhs.gov/schedulerapi/slots/asLocations?limit=100

This time, the API returns a list of 74 objects, numbered 0 through 73. This is less than the maximum number of objects requested, so we can be confident that this is a complete list of locations. We’re making great progress!

Scripting out Requests

Now that we know how to retrieve sites and available appointments via the API, we can write a script to simplify the requests. We’ll create a file called get_ttp_appointments.py and add some logic to retrieve the list of sites using the requests library.

import requests

LOCATIONS_URL = "https://ttp.cbp.dhs.gov/schedulerapi/slots/asLocations?limit=100"

print("Retrieving locations...")
locations = requests.get(LOCATIONS_URL).json()

for loc in locations:
    print("{}, {}: {} ({})".format(
loc['city'], loc['state'], loc['name'].strip(), loc['id']
    ))

Calling .json() on the response object returns native Python list of all the locations. We then iterate through this list, printing the name and numeric ID of each site.

$ python3 ttp1.py 
Retrieving locations...
Mission, TX: Hidalgo Enrollment Center (5001)
San Diego, CA: San Diego -Otay Mesa Enrollment Center (5002)
Brownsville, TX: Brownsville Enrollment Center (5003)
Laredo, TX: Laredo Enrollment Center (5004)
...

Unfortunately, upon inspecting the output, I notice that neither of my desired locations (Richmond and Charlotte) are present in the list. Considering the structure of its URL, it’s fair to assume that this endpoint is returning only locations which have an available appointment. This isn’t going to work.

Let’s try a different approach. Back in the web portal, I select a location with an available appointment, then proceed to schedule an appointment. This brings me to a calendar showing days with available slots. The request inspector reveals a new API endpoint which is used to retrieve available appointments for a specific site:

https://ttp.cbp.dhs.gov/schedulerapi/locations/5444/slots?startTimestamp=2019-07-28T00:00:00&endTimestamp=2019-10-02T00:00:00

This endpoint returns a list of all appointment slots within the requested time span, whether or not they have availability. A time slot is represented as:

{
  "active" : 1,
  "total" : 4,
  "pending" : 0,
  "conflicts" : 0,
  "duration" : 15,
  "timestamp" : "2019-09-30T16:15"
}

Comparing the API data to the calendar API, it appears that the active field indicates the number of available appointments for the time slot.

Recall that our first endpoint (at /schedulerapi/slots?locationId=5444) returned only available appointments, and returned only an empty list for Richmond. But it’s not clear how far into the future the endpoint looks for available appointments. Is it possible to look several months ahead using the calendar endpoint?

We can write a quick script to cycle through all appointment slots roughly one month at a time. (This is called pagination.) We’ll also build in a limiter to stop if we don’t find any appointments in the next year.

from datetime import datetime, timedelta
import requests

TIMESPAN_URL = "https://ttp.cbp.dhs.gov/schedulerapi/locations/14981/slots?startTimestamp={}&endTimestamp={}"

start_time = datetime.now()
start_time.replace(microsecond=0)
results = True
i = 1

while results:
    end_time = start_time + timedelta(days=30)
    url = TIMESPAN_URL.format(start_time.isoformat(), end_time.isoformat())
    slots = requests.get(url).json()
    available_slots = [slot['timestamp'] for slot in slots if slot['active']]
    print("{} - {}: {} slots available".format(start_time, end_time, len(available_slots)))
    if available_slots or i == 12:
        break
    start_time = end_time
    i += 1

Let’s try it out!

$ python3 ttp2.py 
2019-08-06 20:23:22.262813 - 2019-09-05 20:23:22.262813: 0 slots available
2019-09-05 20:23:22.262813 - 2019-10-05 20:23:22.262813: 0 slots available
2019-10-05 20:23:22.262813 - 2019-11-04 20:23:22.262813: 0 slots available
2019-11-04 20:23:22.262813 - 2019-12-04 20:23:22.262813: 0 slots available
2019-12-04 20:23:22.262813 - 2020-01-03 20:23:22.262813: 0 slots available
...

It seems that paging through all time slots doesn’t get us anything more than the first endpoint we tried. To be honest, this was expected, but it’s a great example of a very common API consumption pattern. It could also be useful if you wanted to check for new appointments within specific date ranges.

Ultimately, I’ll have to settle for checking a predetermined list of locations for any available appointments. Here’s what my script ended up looking like:

import requests
import time

APPOINTMENTS_URL = "https://ttp.cbp.dhs.gov/schedulerapi/slots?orderBy=soonest&limit=1&locationId={}&minimum=1"
LOCATION_IDS = {
    'Richmond': 14981,
    'Charlotte': 14321,
}

for city, id in LOCATION_IDS.items():
    url = APPOINTMENTS_URL.format(id)
    appointments = requests.get(url).json()
    if appointments:
        print("{}: Found an appointment at {}!".format(city, appointments[0]['startTimestamp']))
    else:
        print("{}: No appointments available".format(city))
    time.sleep(1)

Note that I've forced the script to sleep for one second in between calls. This is a safety measure to hopefully avoid triggering any abuse mitigation. Normally, public APIs publish their acceptable use policies, but in this case we have no documentation to reference.

Fortunately, a few days later a series of new appointment slots opened up while I was writing this and I was able to nab one. But if you wanted to keep going, it would be easy to extend the script to email you as soon as it finds an open appointment. Just one more example of how automation can put you ahead.