NESO UK Data Portal - Ismena website

info@ismena.com

NESO UK Data Portal

Connector Details

Type

Virtual machines, Single VM , BYOL

Runs on

Google Compute Engine

Last Update

24 October, 2024

Category

Overview

The NESO UK Data Portal Connector provides seamless integration with the National Energy System Operator (NESO) UK Data Portal API, enabling access to energy-related data groups, datasets, and data files. Built on the CKAN platform, this connector acts as a proxy to streamline data retrieval, supporting multiple action points for listing, searching, and querying data. Designed for Google Cloud Platform (GCP), it leverages GCP services like Cloud Functions, Dataflow, or BigQuery for efficient data processing and analysis.

Integration Overview

This document provides a detailed guide for the supported action points, their purpose, configuration, and workflow support using the NESO UK Data Portal Connector on GCP. The connector facilitates programmatic access to energy data, with guidelines to adhere to API rate limits for optimal performance.

Supported Integration Action Points

organizationList: Retrieves a list of all Data Groups (Organizations) on the NESO UK Data Portal.
packageList: Retrieves a list of all Datasets (Packages) available on the Data Portal.
tagList: Retrieves a list of all tags used on the Data Portal.
packageSearch: Searches for datasets matching specified query criteria.
resourceSearch: Searches for data files (resources) matching specified query criteria.
resourceShow: Retrieves metadata for a specific resource by its ID.
packageShow: Retrieves details for a specific dataset by its ID.
datastoreSearch: Retrieves tabular data records from a resource with optional filters.
datastoreSearchSql: Retrieves tabular data records from a resource using a SQL query.

Detailed Integration Documentation

Organization List

Action	organizationList
Purpose	Retrieves a list of all Data Groups (Organizations) available on the NESO UK Data Portal, useful for identifying available data categories.
Parameters	Required: None Optional: None
Configuration	Ensure the connector is configured with the base URL via `CONNECTOR_ENV_NESO_BASE_URL`. Deploy on GCP (Cloud Functions or similar) with rate limits of 1 request per second.
Output	Successful: Returns JSON with `success` (boolean) and `result` (array of Data Group names) Failure: Returns error details (e.g., `{"error": "Invalid request"}`)
Workflow Example	Deploy a Cloud Function with the connector using `CONNECTOR_ENV_NESO_BASE_URL`. Execute organizationList action to fetch all Data Groups. Store results in BigQuery or display in Looker Studio.

Package List

Action	packageList
Purpose	Retrieves a list of all Datasets (Packages) available on the NESO UK Data Portal, enabling discovery of available datasets.
Parameters	Required: None Optional: None
Configuration	Configure the connector with the base URL via `CONNECTOR_ENV_NESO_BASE_URL`. Deploy on GCP (Cloud Functions or App Engine) with a rate limit of 1 request per second.
Output	Successful: Returns JSON with `success` (boolean) and `result` (array of Dataset names) Failure: Returns error details (e.g., `{"error": "Invalid request"}`)
Workflow Example	Set up a Cloud Function to call packageList action. Store dataset list in a GCS bucket for processing. Use Dataflow to filter datasets for specific use cases (e.g., renewable energy).

Tag List

Action	tagList
Purpose	Retrieves a list of all tags used on the NESO UK Data Portal, useful for categorizing and filtering data.
Parameters	Required: None Optional: None
Configuration	Set the base URL via `CONNECTOR_ENV_NESO_BASE_URL`. Deploy on GCP with rate limits set to 1 request per second.
Output	Successful: Returns JSON with `success` (boolean) and `result` (array of tag names, e.g., ["tag1", "tag2"]) Failure: Returns error details (e.g., `{"error": "Invalid request"}`)
Workflow Example	Configure a Cloud Function to execute the tagList action. Store tags in BigQuery to support tag-based filtering in a search interface. Use Looker Studio to visualize tag distributions for data exploration.

Package Search

Action	packageSearch
Purpose	Searches for datasets matching the provided query criteria, enabling targeted dataset discovery.
Parameters	Required: `q` — Query string to search for datasets (string, e.g., "BSUOS") Optional: None
Configuration	Configure the connector with `CONNECTOR_ENV_NESO_BASE_URL`. Deploy on GCP Cloud Functions, adhering to a 1 request per second rate limit.
Output	Successful: Returns JSON with `success` (boolean) and `result` (object containing `count` and `results` array of dataset objects, e.g., [{"id": "dataset1", "title": "BSUOS Dataset"}]) Failure: Returns error details (e.g., `{"error": "Invalid query parameter"}`)
Workflow Example	Deploy a Cloud Function to call packageSearch with `q="BSUOS"`. Store results in BigQuery for analysis. Create a Looker Studio dashboard to display matching datasets.

Resource Search

Action	resourceSearch
Purpose	Searches for data files (resources) matching the provided query criteria, enabling targeted resource discovery.
Parameters	Required: `query` — Query string to search for data files (string, e.g., "name:BSUOS") Optional: None
Configuration	Set `CONNECTOR_ENV_NESO_BASE_URL`. Use GCP Cloud Functions with a 1 request per second rate limit.
Output	Successful: JSON with `success` (boolean) and `result` (object containing `count` and `results` array of resource objects, e.g., [{"id": "resource1", "name": "BSUOS File"}]) Failure: Error details (e.g., `{"error": "Invalid query parameter"}`)
Workflow Example	Execute resourceSearch with `query="name:BSUOS"` via a Cloud Function. Store results in Google Cloud Storage for further processing. Use Dataflow to extract metadata for specific resources.

Resource Show

Action	resourceShow
Purpose	Retrieves metadata for a specific resource by its ID, providing detailed information about a data file.
Parameters	Required: `id` — ID of the resource (string, e.g., "8da765a1-004f-46a5-8b3f-0e5b1787fcb1") Optional: None
Configuration	Configure with `CONNECTOR_ENV_NESO_BASE_URL`. Deploy on GCP with a 1 request per second rate limit.
Output	Successful: JSON with `success` (boolean) and `result` (resource metadata, e.g., {"id": "8da765a1-004f-46a5-8b3f-0e5b1787fcb1", "last_modified": "2023-10-01T12:00:00Z"}) Failure: Error details (e.g., `{"error": "Resource not found"}` for 404)
Workflow Example	Call resourceShow with `id="8da765a1-004f-46a5-8b3f-0e5b1787fcb1"` using a Cloud Function. Store metadata in BigQuery for tracking resource updates. Use the `last_modified` field to trigger Dataflow jobs for updated resources.

Package Show

Action	packageShow
Purpose	Retrieves details for a specific dataset by its ID, providing comprehensive dataset metadata.
Parameters	Required: `id` — ID of the dataset (string, e.g., "embedded-wind-and-solar-forecasts") Optional: None
Configuration	Set `CONNECTOR_ENV_NESO_BASE_URL`. Deploy on GCP with a 1 request per second rate limit.
Output	Successful: JSON with `success` (boolean) and `result` (dataset details, e.g., {"id": "embedded-wind-and-solar-forecasts", "title": "Embedded Wind and Solar Forecasts"}) Failure: Error details (e.g., `{"error": "Resource not found"}` for 404)
Workflow Example	Execute packageShow with `id="embedded-wind-and-solar-forecasts"` via a Cloud Function. Store dataset details in BigQuery for reporting. Visualize dataset metadata in Looker Studio for user access.

Datastore Search

Action	datastoreSearch
Purpose	Retrieves tabular data records from a resource with optional filters, supporting detailed data queries. Requires authorization for private CKAN resources.
Parameters	Required: `resource_id` — ID or alias of the resource (string, e.g., "db6c038f-98af-4570-ab60-24d71ebd0ae5") Optional: `filters, q, full_text, distinct, plain, language, limit, offset, fields, sort, include_total, total_estimation_threshold, records_format`
Configuration	Configure with `CONNECTOR_ENV_NESO_BASE_URL`. Deploy on GCP Cloud Functions or Dataflow, ensuring a 2 requests per minute rate limit. Include authentication headers for private resources.
Output	Successful: JSON with `success` (boolean) and `result` (object containing fields, offset, limit, filters, total, total_was_estimated, records) Failure: Error details (e.g., `{"error": "Unauthorized"}` for 401, `{"error": "Resource not found"}` for 404)
Workflow Example	Deploy a Cloud Function to call datastoreSearch with `resource_id="db6c038f-98af-4570-ab60-24d71ebd0ae5"`, `filters={"column_name": "filter_value"}`, and `records_format=objects`. Store tabular data in BigQuery for analysis. Use Dataflow to process records and generate insights (e.g., energy consumption trends).

Datastore Search SQL

Action	datastoreSearchSql
Purpose	Retrieves tabular data records from a resource using a SQL query, enabling advanced data retrieval. Requires authorization for private CKAN resources.
Parameters	Required: `sql` — SQL query to execute (string, e.g., SELECT * FROM "db6c038f-98af-4570-ab60-24d71ebd0ae5" WHERE "column_name" = 'filter_value') Optional: None
Configuration	Configure with `CONNECTOR_ENV_NESO_BASE_URL`. Deploy on GCP Cloud Functions or Dataflow, adhering to a 2 requests per minute rate limit. Include authentication headers for private resources.
Output	Successful: JSON with `success` (boolean) and `result` (object containing fields, offset, limit, filters, total, total_was_estimated, records) Failure: Error details (e.g., `{"error": "Invalid SQL query"}` for 400, `{"error": "Unauthorized"}` for 401)
Workflow Example	Execute datastoreSearchSql with a SQL query via a Cloud Function (e.g., SELECT * FROM "db6c038f-98af-4570-ab60-24d71ebd0ae5" WHERE "column_name" = 'filter_value'). Load results into BigQuery for querying and analysis. Use Looker Studio to visualize results or trigger Dataflow for advanced processing.

Workflow Creation with the Connector

Example Workflow: Energy Data Retrieval and Analysis

List Available Datasets	Execute the packageList action to fetch all available datasets using a Cloud Function. Store dataset names in BigQuery to create a catalog. Display the catalog in a Looker Studio dashboard for user selection.
Search for Specific Datasets	Execute the packageSearch action with `q="wind energy"` to find relevant datasets. Process the response to extract dataset IDs and titles, storing them in Google Cloud Storage. Use Dataflow to filter datasets related to renewable energy for further analysis.
Retrieve Resource Metadata	Execute the resourceShow action with a specific resource ID (e.g., `id="8da765a1-004f-46a5-8b3f-0e5b1787fcb1"`) to fetch metadata. Store metadata in BigQuery, using the `last_modified` field to track updates. Trigger a Cloud Scheduler job to periodically check for updates using resourceShow.
Query Tabular Data	Execute the datastoreSearch action with `resource_id="db6c038f-98af-4570-ab60-24d71ebd0ae5"`, `filters={"type": "wind"}`, and `records_format=objects` to fetch tabular data. Load results into BigQuery for querying and analysis. Use Dataflow to process records and compute metrics (e.g., average energy output).
Advanced SQL Query	Execute the datastoreSearchSql action with a SQL query (e.g., `SELECT * FROM "db6c038f-98af-4570-ab60-24d71ebd0ae5" WHERE "type" = 'solar'`) to fetch specific data. Store results in BigQuery and use Looker Studio to visualize trends (e.g., solar energy production by region). Schedule recurring queries with Cloud Scheduler for continuous monitoring.

Pricing

bill-and-calculator-2023-11-27-05-25-06-utc 1

Request a Quote

Support

For Technical support please contact us on

custom-connectors-support@isolutions.sa

A tech solution company dedicated to providing innovation thus empowering businesses to thrive in the digital age.

Register to IBM x iSolution Event