AbstractHttpServiceHandler throws Java heap space exception on receiving large http put file
Description
Release Notes
blocks
Activity
Jeff Dix December 23, 2015 at 9:30 PM
I was actually on the develop branch. I move to the release 3.3 branch and I see the API now.
Andreas Neumann December 21, 2015 at 6:17 PM
, it is strange that you do not see it... are you using the release/3.3 branch?
The API did change: The existing DatasetContext was replaced by DatasetProvider, which extends DatasetContext and adds two new methods. All program contexts, including HttpServiceContext, implement this.
Jeff Dix December 21, 2015 at 2:35 PM
Did the API change, I do not see discardDataset as part of the HttpServiceContext like the example shows?
Andreas Neumann December 19, 2015 at 1:41 AM
Oh, and discardDataset() does not close the transaction. It only indicates to the system that this dataset can be discarded after the current transaction finishes.
Andreas Neumann December 19, 2015 at 1:40 AM
This was added in 3.3, in a new API called DatasetProvider that will replace DatasetContext.
Its purpose is the opposite of getDataset(): to return the dataset to the system, because you do not need it any longer, and to allow the system to free resources that may be used by this dataset. Calling it is not necessary: if you expect that you will be using the same dataset again and again, it is better not to call it. In that case, the system will keep the dataset in a cache after the current transaction finishes, and the next time you call getDataset() for the same dataset, it can return the cached instance to you.
I am extending AbstractHttpServiceHandler and have implemented a put endpoint. Right now the only think the endpoint does is log it has been called.
public static class LargeFileHandler extends AbstractHttpServiceHandler {
private static final Logger logger = LoggerFactory.getLogger(LargeFileHandler.class);
@PUT
@Path("file")
public void ingest(HttpServiceRequest request,
HttpServiceResponder responder) {
logger.info("Ingest called");
//ByteBuffer bb = request.getContent();
}
}
I am uploading a file to the endpoint using curl with the --data-binary option like the example below.
curl -w'\n' -v -X PUT -H "Content-Type: application/octet-stream" --data-binary @file http://host:10000/v3/namespaces/default/apps/LargeFileApp/services/LargeFileService/methods/file
In the service handler I am using request.getContent() to access the file received by the service handler. For small files this works and the contents of the file is in accessible in the ByteBuffer returned by request.getContent(). When I upload a file larger than 500MB I get the exception below before my service handler method is called.
2015-09-10 15:28:07,908 - ERROR [New I/O worker #1:c.c.h.RequestRouter@141] - Exception caught in channel processing.
java.lang.OutOfMemoryError: Java heap space
at org.jboss.netty.buffer.HeapChannelBuffer.<init>(HeapChannelBuffer.java:42) ~[io.netty.netty-3.6.6.Final.jar:na]
at org.jboss.netty.buffer.BigEndianHeapChannelBuffer.<init>(BigEndianHeapChannelBuffer.java:34) ~[io.netty.netty-3.6.6.Final.jar:na]
at org.jboss.netty.buffer.ChannelBuffers.buffer(ChannelBuffers.java:134) ~[io.netty.netty-3.6.6.Final.jar:na]
at org.jboss.netty.buffer.HeapChannelBufferFactory.getBuffer(HeapChannelBufferFactory.java:68) ~[io.netty.netty-3.6.6.Final.jar:na]
at org.jboss.netty.buffer.CompositeChannelBuffer.copy(CompositeChannelBuffer.java:568) ~[io.netty.netty-3.6.6.Final.jar:na]
at org.jboss.netty.buffer.AbstractChannelBuffer.copy(AbstractChannelBuffer.java:522) ~[io.netty.netty-3.6.6.Final.jar:na]
at org.jboss.netty.handler.codec.http.HttpChunkAggregator.appendToCumulation(HttpChunkAggregator.java:208) ~[io.netty.netty-3.6.6.Final.jar:na]
at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:175) ~[io.netty.netty-3.6.6.Final.jar:na]
at co.cask.http.RequestRouter.messageReceived(RequestRouter.java:79) ~[co.cask.http.netty-http-0.12.0.jar:na]
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) ~[io.netty.netty-3.6.6.Final.jar:na]
at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) ~[io.netty.netty-3.6.6.Final.jar:na]
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) ~[io.netty.netty-3.6.6.Final.jar:na]
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) ~[io.netty.netty-3.6.6.Final.jar:na]
at org.jboss.netty.handler.codec.http.HttpContentEncoder.messageReceived(HttpContentEncoder.java:69) ~[io.netty.netty-3.6.6.Final.jar:na]
at co.cask.http.NettyHttpService$2.handleUpstream(NettyHttpService.java:220) ~[co.cask.http.netty-http-0.12.0.jar:na]
I am running this test on a cluster. When I run on the SDK I am not able to upload a file larger than 157286400 bytes and I get the following exception:
Exception Encountered while processing request: HTTP content length exceeded 157286400 bytes.