Connector Details
Type
Virtual machines, Single VM , BYOL
Runs on
Google Compute Engine
Last Update
24 October, 2024
Category
Big data, Analytics, DevOps
Qatar Open Data Portal Connector
Overview
The Qatar Open Data Connector enables seamless integration with the Qatar Open Data Portal, providing access to catalog and dataset-related data for applications such as data analytics, visualization tools, or geographic information systems. The connector acts as a proxy to streamline data retrieval, supporting actions for dataset enumeration, record queries, data exports, and facet-based navigation.
Integration Overview
This document outlines each integration point, its purpose, configuration, and workflow support using the Qatar Open Data Connector.
Supported Integration Action Points
- Catalog Datasets Retrieval: Retrieves a list of all datasets in the catalog
- Catalog Exports Listing: Lists available export formats for the dataset catalog.
- Catalog Export by Format : Exports the dataset catalog in a specified format (e.g., CSV, JSON).
- Catalog CSV Export: Exports the dataset catalog as CSV with customizable parameters.
- Catalog DCAT Export: Exports catalog metadata in DCAT-AP format (RDF/XML).
- Catalog Facets Retrieval: Retrieves global catalog-level facets for guided navigation.
- Dataset Metadata Retrieval: Retrieves metadata for a specific dataset.
- Dataset Records Retrieval: Retrieves records for a specific dataset.
- Dataset Record Retrieval: Retrieves a specific record from a dataset.
- Dataset Exports Listing: Lists supported export formats for a dataset.
- Dataset Export by Format : Exports a dataset in a specific format (e.g., CSV, JSON).
- Dataset CSV Export : Exports a dataset in CSV format with specific parameters.
- Dataset Parquet Export : Exports a dataset in Parquet format.
- Dataset GPX Export : Exports a dataset in GPX format.
- Dataset Facets Retrieval: Retrieves available facets for a dataset.
- Dataset Attachments Retrieval: Retrieves files or attachments related to a dataset.
Detailed Integration Documentation
Catalog Datasets Retrieval
Action
getDatasets
Purpose
Retrieves a comprehensive list of all datasets in the catalog, serving as the primary entry point for exploring available data.
Parameters
- Required: None.
- Optional:
- select: Specify fields to include (e.g., dataset_id, fields).
- where: Filter datasets using ODSQL (e.g., publisher=”Planning and Statistics Authority”).
- order_by: Sort results (e.g., modified desc).
- limit: Number of items (default: 10, max: 100).
- offset: Starting index (default: 0).
- refine: Filter by facet (e.g., publisher:Planning and Statistics Authority).
- exclude: Exclude facet values (e.g., publisher:Planning and Statistics Authority).
- lang: Language for formatting (e.g., en).
- timezone: Timezone for datetime fields (e.g., Asia/Qatar).
- group_by: Group results (e.g., publisher).
- include_links: Include HATEOAS links (boolean, default: false).
- include_app_metas: Include metadata (boolean, default: false).
Configuration
Ensure the connector is configured with the appropriate environment variables for authentication and connectivity.
Output
- Successful: Returns a JSON object with total_count, _links, and results (array of dataset objects containing dataset_id, fields, metas, etc.).
- Failure: Returns error details (e.g., invalid parameters).
Workflow Example
1. Configure the connector for the Qatar Open Data Portal.
2. Execute the Catalog Datasets Retrieval action to fetch a list of datasets.
3. Process the response to identify datasets (e.g., population-data) for further actions.
Catalog Exports Listing
Action
listExportFormats
Purpose
Lists all available export formats for the dataset catalog, helping users choose an appropriate format for data export.
Parameters
- Required: None.
- Optional: None.
Configuration
Ensure the connector is configured with the appropriate environment variables for authentication and connectivity.
Output
- Successful: Returns a JSON object with a links array listing export format options (e.g., csv, json).
- Failure: Returns error details (e.g., upstream API error).
Workflow Example
1. Execute the Catalog Exports Listing action to retrieve available export formats.
2. Review the response to select a format (e.g., csv) for catalog export.
3. Proceed to the Catalog Export by Format action for export.
Catalog Export by Format
Action
exportDatasets
Purpose
Exports the entire dataset catalog in a user-specified format, such as JSON or CSV, for versatile data use.
Parameters
- Required: format (e.g., csv, json).
- Optional:
- select: Specify fields to include (e.g., dataset_id, fields).
- where: Filter datasets using ODSQL (e.g., publisher=”Planning and Statistics Authority”).
- order_by: Sort results (e.g., modified desc).
- limit: Number of items (default: -1, retrieves all records).
- offset: Starting index (default: 0).
- refine: Filter by facet (e.g., publisher:Planning and Statistics Authority).
- exclude: Exclude facet values (e.g., publisher:Planning and Statistics Authority).
- lang: Language for formatting (e.g., en).
- timezone: Timezone for datetime fields (e.g., Asia/Qatar).
- group_by: Group results (e.g., publisher).
Configuration
Ensure the connector is configured with the appropriate environment variables for authentication and connectivity.
Output
- Successful: Returns a file in the specified format.
- Failure: Returns error details (e.g., unsupported format).
Workflow Example
1. Use the Catalog Exports Listing action to identify supported formats.
2. Execute the Catalog Export by Format action with format=csv and limit=10.
3. Save the exported CSV file for external analysis.
Catalog CSV Export
Action
exportCatalogCSV
Purpose
Parameters
- Required: None.
- Optional:
- delimiter: Field delimiter (e.g., ,, default: ;).
- list_separator: Multivalued string separator (default: ,).
- quote_all: Quote all strings (boolean, default: false).
- with_bom: Include Unicode BOM (boolean, default: true).
- select: Specify fields to include (e.g., dataset_id, fields).
- where: Filter datasets using ODSQL (e.g., publisher=”Planning and Statistics Authority”).
- order_by: Sort results (e.g., modified desc).
- limit: Number of items (default: -1, retrieves all records).
- offset: Starting index (default: 0).
- refine: Filter by facet (e.g., publisher:Planning and Statistics Authority).
- exclude: Exclude facet values (e.g., publisher:Planning and Statistics Authority).
- lang: Language for formatting (e.g., en).
- timezone: Timezone for datetime fields (e.g., Asia/Qatar).
- group_by: Group results (e.g., publisher).
Configuration
Ensure the connector is configured with the appropriate environment variables for authentication and connectivity.
Output
- Successful: Returns a CSV file.
- Failure: Returns error details (e.g., invalid parameters).
Workflow Example
1. Execute the Catalog CSV Export action with delimiter=,.
2. Save the CSV file containing catalog metadata.
3. Use the file for data analysis or reporting.
Catalog DCAT Export
Action
exportCatalogDCAT
Purpose
Exports catalog metadata in RDF/XML (DCAT-AP) format, enabling integration with metadata systems supporting DCAT standards.
Parameters
- Required: dcat_ap_format (e.g., _ap_ch, _ap_de).
- Optional:
- include_exports: Export formats to include (e.g., csv,json).
- use_labels_in_exports: Use field labels (boolean, default: true).
Configuration
Ensure the connector is configured with the appropriate environment variables for authentication and connectivity.
Output
- Successful: Returns an RDF/XML file.
- Failure: Returns error details (e.g., invalid DCAT format).
Workflow Example
1. Execute the Catalog DCAT Export action with dcat_ap_format=_ap_ch.
2. Save the RDF/XML file.
3. Use the file for metadata integration with DCAT-compatible systems.
Catalog Facets Retrieval
Action
getDatasetsFacets
Purpose
Retrieves facet values for datasets to aid navigation, helping users filter datasets based on attributes like publisher.
Parameters
- Required: None.
- Optional:
- facet: Specify facet to retrieve (e.g., publisher).
- where: Filter datasets using ODSQL (e.g., publisher=”Planning and Statistics Authority”).
- refine: Filter by facet (e.g., publisher:Planning and Statistics Authority).
- exclude: Exclude facet values (e.g., publisher:Planning and Statistics Authority).
- timezone: Timezone for datetime fields (e.g., Asia/Qatar).
Configuration
Ensure the connector is configured with the appropriate environment variables for authentication and connectivity.
Output
- Successful: Returns a JSON object with links and facets arrays.
- Failure: Returns error details (e.g., invalid facet).
Workflow Example
1. Execute the Catalog Facets Retrieval action with facet=publisher.
2. Review facet values (e.g., Planning and Statistics Authority) to refine dataset queries.
3. Use facets to filter results in subsequent actions.
Dataset Metadata Retrieval
Action
getDataset
Purpose
Retrieves detailed metadata for a specific dataset, including fields and endpoints to plan further data queries or exports
Parameters
- Required: dataset_id (e.g., population-data).
- Optional:
- select: Specify fields to include (e.g., dataset_id, fields).
- lang: Language for formatting (e.g., en).
- timezone: Timezone for datetime fields (e.g., Asia/Qatar).
- include_links: Include HATEOAS links (boolean, default: false).
- include_app_metas: Include metadata (boolean, default: false).
Configuration
Ensure the connector is configured with the appropriate environment variables for authentication and connectivity.
Output
- Successful: Returns a JSON object with dataset details (e.g., dataset_id, fields, metas).
- Failure: Returns error details (e.g., invalid dataset_id).
Workflow Example
1. Use the Catalog Datasets Retrieval action to identify a dataset (e.g., population-data).
2. Execute the Dataset Metadata Retrieval action with the selected dataset_id.
3. Review metadata to plan record queries or exports.
Dataset Records Retrieval
Action
getRecords
Purpose
Queries records from a specific dataset, allowing users to access detailed data entries for analysis or visualization.
Parameters
- Required:dataset_id.
- Optional:
- select: Specify fields to include (e.g., municipality, population).
- where: Filter datasets using ODSQL (e.g., year=2020).
- group_by: Group results (e.g., municipality).
- order_by: Sort results (e.g., population desc).
- limit: Number of items (default: 10, max: 100).
- offset: Starting index (default: 0).
- refine: Filter by facet (e.g., year:2020).
- exclude: Exclude facet values (e.g., year:2020).
- lang: Language for formatting (e.g., en).
- timezone: Timezone for datetime fields (e.g., Asia/Qatar).
- include_links: Include HATEOAS links (boolean, default: false).
- include_app_metas: Include metadata (boolean, default: f
Configuration
Ensure the connector is configured with the appropriate environment variables for authentication and connectivity.
Output
- Successful: Returns a JSON object with total_count, _links, and results (array of records).
- Failure:Returns error details (e.g., invalid dataset_id).
Workflow Example
1. Use the Dataset Metadata Retrieval action to select a dataset (e.g., population-data).
2. Execute the Dataset Records Retrieval action with dataset_id=population-data and where=year:2020.
3. Process the response to display or analyze records.
Dataset Record Retrieval
Action
getRecord
Purpose
Retrieves a single record from a dataset, providing detailed information for a specific data entry.
Parameters
- Required:dataset_id, record_id (e.g., pop_001).
- Optional:
- select: Specify fields to include (e.g., municipality, population).
- lang: Language for formatting (e.g., en).
- timezone: Timezone for datetime fields (e.g., Asia/Qatar).
Configuration
Ensure the connector is configured with the appropriate environment variables for authentication and connectivity.
Output
- Successful: Returns a JSON object representing the record.
- Failure: Returns error details (e.g., invalid record_id).
Workflow Example
1. Use the Dataset Records Retrieval action to identify a record in population-data.
2. Execute the Dataset Record Retrieval action with dataset_id=population-data and record_id=pop_001.
3. Review the record details for further processing.
Dataset Exports Listing
Action
listDatasetExportFormats
Purpose
Lists available export formats for a specific dataset, helping users select a format for dataset export.
Parameters
- Required:dataset_id.
- Optional:
- None.
Configuration
Ensure the connector is configured with the appropriate environment variables for authentication and connectivity.
Output
- Successful: Returns a JSON object with a links array listing export formats.
- Failure: Returns error details (e.g., invalid dataset_id).
Workflow Example
1. Use the Dataset Metadata Retrieval action to select a dataset (e.g., population-data).
2. Execute the Dataset Exports Listing action with dataset_id=population-data.
3. Review available formats (e.g., csv, json) for export.
Dataset Export by Format
Action
exportRecords
Purpose
Exports a dataset in a user-specified format, such as CSV or JSON, for diverse applications.
Parameters
- Required:dataset_id, format (e.g., csv, json).
- Optional:
- select: Specify fields to include (e.g., municipality, population).
- where: Filter datasets using ODSQL (e.g., year=2020).
- order_by: Sort results (e.g., population desc).
- group_by: Group results (e.g., municipality).
- limit: Number of items (default: -1, retrieves all records).
- refine: Filter by facet (e.g., year:2020).
- exclude: Exclude facet values (e.g., year:2020).
- lang: Language for formatting (e.g., en).
- timezone: Timezone for datetime fields (e.g., Asia/Qatar).
- use_labels: Use field labels (boolean, default: true).
- epsg: Coordinate system for geospatial data (e.g., 4326).
Configuration
Ensure the connector is configured with the appropriate environment variables for authentication and connectivity.
Output
- Successful: Returns a file in the specified format.
- Failure: Returns error details (e.g., invalid format).
Workflow Example
1. Use the Dataset Exports Listing action to identify supported formats for population-data.
2. Execute the Dataset Export by Format action with dataset_id=population-data and format=csv.
3. Save the exported file for external use.
Dataset CSV Export
Action
exportRecordsCSV
Purpose
Exports a dataset in CSV format with customizable parameters, enabling tailored data extraction for specific needs.
Parameters
- Required:dataset_id.
- Optional:
- delimiter: Field delimiter (e.g., ,, default: ;).
- list_separator: Multivalued string separator (default: ,).
- quote_all: Quote all strings (boolean, default: false).
- with_bom: Include Unicode BOM (boolean, default: true).
- select: Specify fields to include (e.g., municipality, population).
- where: Filter datasets using ODSQL (e.g., year=2020).
- order_by: Sort results (e.g., population desc).
- group_by: Group results (e.g., municipality).
- limit: Number of items (default: -1, retrieves all records).
- refine: Filter by facet (e.g., year:2020).
- exclude: Exclude facet values (e.g., year:2020).
- lang: Language for formatting (e.g., en).
- timezone: Timezone for datetime fields (e.g., Asia/Qatar).
- use_labels: Use field labels (boolean, default: true).
Configuration
Ensure the connector is configured with the appropriate environment variables for authentication and connectivity.
Output
- Successful:Returns a CSV file.
- Failure:Returns error details (e.g., invalid parameters).
Workflow Example
1. Execute the Dataset CSV Export action with dataset_id=population-data and delimiter=,.
2. Save the CSV file containing dataset records.
3. Use the file for data analysis or reporting.
Dataset Parquet Export
Action
exportRecordsParquet
Purpose
Exports a dataset in Parquet format, ideal for efficient storage and processing in data analytics platforms.
Parameters
- Required:dataset_id.
- Optional:
- parquet_compression: Compression type (snappy or zstd, default: snappy).
- select: Specify fields to include (e.g., municipality, population).
- where: Filter datasets using ODSQL (e.g., year=2020).
- order_by: Sort results (e.g., population desc).
- group_by: Group results (e.g., municipality).
- limit: Number of items (default: -1, retrieves all records).
- refine: Filter by facet (e.g., year:2020).
- exclude: Exclude facet values (e.g., year:2020).
- lang: Language for formatting (e.g., en).
- timezone: Timezone for datetime fields (e.g., Asia/Qatar).
- use_labels: Use field labels (boolean, default: true).
Configuration
Ensure the connector is configured with the appropriate environment variables for authentication and connectivity.
Output
- Successful:Returns a Parquet file.
- Failure:Returns error details (e.g., invalid dataset_id).
Workflow Example
1. Execute the Dataset Parquet Export action with dataset_id=population-data.
2. Save the Parquet file.
3. Use the file for data processing in compatible systems.
Dataset GPX Export
Action
exportRecordsGPX
Purpose
Exports a dataset in GPX format, suitable for geographic data visualization and GPS applications.
Parameters
- Required:dataset_id.
- Optional:
- name_field: Field for waypoint names (e.g., municipality).
- description_field_list: Fields for waypoint descriptions (e.g., population).
- use_extension: Include extensions in GPX (boolean, default: true).
- select: Specify fields to include (e.g., municipality, population).
- where: Filter datasets using ODSQL (e.g., year=2020).
- order_by: Sort results (e.g., population desc).
- group_by: Group results (e.g., municipality).
- limit: Number of items (default: -1, retrieves all records).
- refine: Filter by facet (e.g., year:2020).
- exclude: Exclude facet values (e.g., year:2020).
- lang: Language for formatting (e.g., en).
- timezone: Timezone for datetime fields (e.g., Asia/Qatar).
- use_labels: Use field labels (boolean, default: true).
- epsg: Coordinate system for geospatial data (e.g., 4326).
Configuration
Ensure the connector is configured with the appropriate environment variables for authentication and connectivity.
Output
- Successful:Returns a GPX file.
- Failure:Returns error details (e.g., invalid parameters).
Workflow Example
1. Execute the Dataset GPX Export action with dataset_id=population-data and name_field=municipality.
2. Save the GPX file.
3. Use the file for geographic data visualization.
Dataset Facets Retrieval
Action
getRecordsFacets
Purpose
Retrieves facet values for records in a specific dataset, aiding in filtering and navigating dataset records effectively.
Parameters
- Required:dataset_id.
- Optional:
- facet: Specify facet to retrieve (e.g., year).
- where: Filter datasets using ODSQL (e.g., year=2020).
- refine: Filter by facet (e.g., year:2020).
- exclude: Exclude facet values (e.g., year:2020).
- lang: Language for formatting (e.g., en).
- timezone: Timezone for datetime fields (e.g., Asia/Qatar).
Configuration
Ensure the connector is configured with the appropriate environment variables for authentication and connectivity.
Output
- Successful:Returns a JSON object with links and facets arrays.
- Failure:Returns error details (e.g., invalid facet).
Workflow Example
1. Use the Dataset Metadata Retrieval action to select population-data.
2. Execute the Dataset Facets Retrieval action with dataset_id=population-data and facet=year.
3. Use facet values to refine record queri
Dataset Attachments Retrieval
Action
getDatasetAttachments
Purpose
Retrieves files or attachments related to a specific dataset, providing access to supplementary data or resources.
Parameters
- Required:dataset_id.
- Optional:
- None.
Configuration
Ensure the connector is configured with the appropriate environment variables for authentication and connectivity.
Output
- Successful:Returns a JSON object with links and attachments arrays (e.g., href, mime-type, title).
- Failure:Returns error details (e.g., invalid dataset_id).
Workflow Example
1. Use the Dataset Metadata Retrieval action to select population-data.
2. Execute the Dataset Attachments Retrieval action with dataset_id=population-data.
3. Download attachments (e.g., population_report.pdf) for further use.
Workflow Creation with the Connector
Example Workflow: Exploring and Exporting Population Statistics
Retrieve Catalog Datasets:
- Use the Catalog Datasets Retrieval action to fetch a list of available datasets.
- Identify the target dataset (e.g., population-data).
Refine Dataset Exploration:
Execute the Dataset Facets Retrieval action with dataset_id=population-data and facet=year to filter records.
Query Dataset Records:
Use the Dataset Records Retrieval action with dataset_id=population-data and where=year:2020 to fetch relevant records.
Export Dataset Data:
- Encode the dataset in CSV format using the Dataset CSV Export action with dataset_id=population-data.
- Export the dataset in Parquet format via the Dataset Parquet Export action for analytical processing.
- Generate a GPX file for geographic visualization using the Dataset GPX Export action with name_field=municipality.
Retrieve Metadata and Attachments:
- Fetch metadata for population-data using the Dataset Metadata Retrieval action.
- Download attachments (e.g., reports) via the Dataset Attachments Retrieval action.
Export Catalog Metadata
- Export the catalog as CSV using the Catalog CSV Export action.
- Generate DCAT-AP metadata via the Catalog DCAT Export action with dcat_ap_format=_ap_ch.
Pricing
iSolution logo - white - transparent 250 px