Too many open files

Description

Using DBTableToHBaseTable adapter example from cdap-guides caused 'java.lang.OutOfMemoryError: PermGen space'. This is a known issue.

Increasing MxPermSize from 128m to 512m and running the same example adapter caused a 'too many files open' error. This came about after running a few simple tests, then leaving the system alone for a while (hours).

lsof on the cdap standalone process showed over 7700 open files (list attached).

Over 6000 files are named: /opt/cdap/cdap-sdk-3.0.0/data/tmp/<13-digit-number><number>/<13-digit-number><number>/lib/*

Of those there are 134 duplicates of 46 jar files split between the file, and a '(deleted)' version of that same file.

Release Notes

None

Activity

Show:
Bhooshan Mogal
June 16, 2015, 1:42 AM

I was able to reproduce this issue within 30 mins of leaving the database to hbase table adapter running with MySQL. However, I cannot reproduce the issue with a postgresql source.

From the hprof that I got from as well as the one from my setup, it looks like there is an AbandonedConnectionCleanupThread that is not destroyed, which seems to hold a reference to the program classloader, in turn causing it to not cleanup the tmp directory.

Related: http://stackoverflow.com/questions/11872316/tomcat-guice-jdbc-memory-leak

Investigating a fix, will update once it is done.

Bhooshan Mogal
June 16, 2015, 6:47 PM
Bhooshan Mogal
June 16, 2015, 8:01 PM

I applyied the above PR and ran a test with DBAdapter on an ubuntu box. The logs indicated that cleanup of that dangling thread was successful each time. However, I started seeing OutOfMemory errors again after about an hour. I see that the data/tmp directory has some class files that did not get cleaned up.

However, this looks like a different issue from before. Unable to attach jmap to the process this time. However, attaching a cdap log file that contains a thread dump at the end.

Sreevatsan Raman
June 20, 2015, 6:02 PM
Bhooshan Mogal
June 23, 2015, 2:51 AM
Fixed

Assignee

Bhooshan Mogal

Reporter

Dave Finnegan

Labels

None

Docs Impact

None

UX Impact

None

Components

Fix versions

Affects versions

Priority

Major
Configure