Streaming HTTP handlers

Services can be used for ingest and egress of data. In current CDAP (3.2.0), however, there are limitations to what you can do:

• Every method call of a service handler is executed in a transaction. The typical transaction timeout is configured at around 30 seconds. That means, if the handler methods needs longer than that to complete, the transaction will fail.

• The content of the HTTP request is always buffered up in memory, hence the handler cannot receive large data. It would be better to stream the content. 

• In case of transaction conflicts, the handler has no control over handling that error. 

Here are some use cases where these limitations get in the way:

1. A service handler to upload partitions to a partitioned file set:

   • With each request, a large file is received. 

   • Meta data about the file is received in the HTTP headers

   • Based on the meta data, the handler determines the partition key for the file

   • The content of the request is consumed and streamed to a file

   • The handler validates the file (possible using a checksum, or validating its size or number of records)

   • The handler may also parse the content as it is streamed and validate it using lookups in a dataset. 

   • The handler registers the file as a new partition

   • If an error occurs in any of these steps, the file must be deleted, or moved to a quarantine area; possibly a record of the error needs to be saved to a dataset

   • If there is a transaction conflict, the same applies. 

   • Also, in case of an error, the handler has control over the HTTP response

2. A service handler to download large files:

   • Similar to 1., with the exception that this is simpler because no writes happen (and no conflicts) 

   • Also, the request is small but the response may be very large and take a long time to send.

3. A handler to receive a sequence of records, and to process them one by one

   • Processing a record may mean storing it in a dataset, or lookup in a dataset

   • The response may indicate how many records were successfully processed (some may have conflicts)

   • The response may contain a new record for every record received.

   • The processing should continue in case of an error (even a transaction conflict). 

   • Possibly each record must be processed in its own transaction