Introduction 

Open Datasets from Federal, State, and Municipal Government agencies have largely standardized on Socrata's platform to expose data to developers.  Socrata has an HTTP REST API known as SODA for querying and filtering data from a given dataset.   SODA includes simple filter and a complex query language available over http known as SoQL (not to be confused with salesforce object query language with the same abbreviation).  Extending the capabilities of Hydrator Pipelines to include freely available government Datasets will open new use cases.  

User Stories

• As a Hydrator User I want to use the SODA HTTP REST API to query specific open datasets and use simple filters to retrieve information so that I can include it in my Hydrator Pipelines
• As a Hydrator User I want to use the SODA HTTP REST API to issue a SQL-like query to query a specific open dataset to retrieve information so that I can include it in my Hydrator Pipelines
• As a Hydrator User I want to use my SODA application token when connecting to an open dataset so that my throughput can be improved.
• As a Hydrator User I want the ability to disable pagination in queries I issue to the SODA HTTP REST API so that I can retrieve data without pagination restrictions.  
  
  

Example

Example for how the plugin should work

Implementation Tips

• SODA API: Documentation
• SODA API Filtering: Documentation
• SODA API SoQL (SQL-Like Query Language) Queries: Documentation
• SODA API application throughput limitations: Documentation

Design 

Design: