{"id":3584,"date":"2023-04-12T15:39:33","date_gmt":"2023-04-12T15:39:33","guid":{"rendered":"https:\/\/quebit.com\/askquebit\/gather-data-from-an-api-using-the-script-source-node\/"},"modified":"2023-04-12T15:39:33","modified_gmt":"2023-04-12T15:39:33","slug":"gather-data-from-an-api-using-the-script-source-node","status":"publish","type":"post","link":"https:\/\/quebit.com\/askquebit\/gather-data-from-an-api-using-the-script-source-node\/","title":{"rendered":"Gather data from an API using the Script source node"},"content":{"rendered":"

The following article outlines how Euclid Studio can be used to request data from an API and process it so it can be used in the system. This requires some knowledge of Python coding that is used within the Script source node and how APIs function.<\/p>\n

APIs usually have associated documentation that describes how to access them. Most APIs have an associated Authentication or API Key that is needed to query the information.<\/p>\n

This example uses the OpenWeather API to get historical weather information for a specific day and location. To be able to access this specific API, only a free OpenWeather account is needed. Once an account is made, an API Key is provided to the account. More information on OpenWeather APis can be found here: https:\/\/openweathermap.org\/api<\/a><\/p>\n

\"\"<\/p>\n

In Euclid Studio, add a Script source node to the flow. Once added, open the node, change the Script Type to Python, and click Rebuild.<\/p>\n

\"\"<\/p>\n

With the node set to use a Python script, the python code can be entered. This code will vary for each API, as the data received from each would differ. Refer to an API\u2019s documentation for more information. No matter the API used, there are key parts to the code that are needed. The full example code is available at the end of the document.<\/p>\n

At the top of the code, it is required to import the needed packages for the script to use. The requests, json, and pandas packages are required for this to work. In my example, I am also using the datetime package to include a needed date conversion.<\/p>\n

\"\"<\/p>\n

Directly below this, the APIKEY, LAT, and LON variables are set. The LAT and LON in the example are the Latitude and Longitude that will be used in the API request to determine what location of weather data to request. The APIKEY variable is populated with the personal APIKEY associated with an OpenWeather account. These variables will be passed into the API request, and could be replaced with Euclid Studio parameters if desired. This information is obfuscated in the full code at the end of this document.<\/p>\n

Then the main function is defined. In the main function, a \u201cyesterday_unix_timestamp\u201d variable is set based on the current day, as the OpenWeather API required a unix timestamp to determine what day of data to return.<\/p>\n

\"\"<\/p>\n

Then, the requests package is used to send a GET request to the OpenWeather API. In this request, there are variables used to populate the information that the API needs to determine what data to send as a result. To ensure the result usable by python, the json package is used to convert the JSON result into a python object.<\/p>\n

\"\"<\/p>\n

The JSON result returns a lot of information. To keep this example simple, only two pieces of information from the result are going to be used. These are the Date in local time, and the Temperature at that time. To prepare to store this data, empty lists are created to store the date and temperature.<\/p>\n

\"\"<\/p>\n

Now these lists need to be populated by the information returned from the API request. Finding the exact information can be difficult based on the structure of the data that the API returns. In this example, the JSON returns a list that contains weather information for each hour of the requested day. Knowing this, the code should look at the information for each hour and then add that to the currently empty lists. For the date_local list, the returned unix timestamp is first converted to a human readable string before adding it. The temperature is added to the list without alteration.<\/p>\n

\"\"<\/p>\n

Now that the lists are populated with 24 hours of data, they need to be combined into a python dictionary.<\/p>\n

\"\"<\/p>\n

Then, using the pandas package, the dictionary is transformed into a dataframe with defined columns. This dataframe is then exported in a csv file in the desired location.<\/p>\n

\"\"<\/p>\n

At the very end of the code, the main() function is called.<\/p>\n

When previewing the results of the code the expected 24 rows of data are returned. Don\u2019t worry about the high temperatures, the API provides the information in Kelvin. The hours also show information from different days due to timezone offsets, as the requested data is provided using UTC.<\/p>\n

\"\"<\/p>\n

Now that the desired information is being output to a csv file, it can be used as needed with a Flat File source node.<\/p>\n

Aiming for simplicity in this explanation, the example lacks error catching that should be present in a production script. Some common errors that should be checked for and considered in the script include the following:<\/p>\n