Using DBTableToHBaseTable adapter example from cdap-guides caused 'java.lang.OutOfMemoryError: PermGen space'. This is a known issue.
Increasing MxPermSize from 128m to 512m and running the same example adapter caused a 'too many files open' error. This came about after running a few simple tests, then leaving the system alone for a while (hours).
lsof on the cdap standalone process showed over 7700 open files (list attached).
Over 6000 files are named: /opt/cdap/cdap-sdk-3.0.0/data/tmp/<13-digit-number><number>/<13-digit-number><number>/lib/*
Of those there are 134 duplicates of 46 jar files split between the file, and a '(deleted)' version of that same file.
I was able to reproduce this issue within 30 mins of leaving the database to hbase table adapter running with MySQL. However, I cannot reproduce the issue with a postgresql source.
From the hprof that I got from as well as the one from my setup, it looks like there is an AbandonedConnectionCleanupThread that is not destroyed, which seems to hold a reference to the program classloader, in turn causing it to not cleanup the tmp directory.
Investigating a fix, will update once it is done.
I applyied the above PR and ran a test with DBAdapter on an ubuntu box. The logs indicated that cleanup of that dangling thread was successful each time. However, I started seeing OutOfMemory errors again after about an hour. I see that the data/tmp directory has some class files that did not get cleaned up.
However, this looks like a different issue from before. Unable to attach jmap to the process this time. However, attaching a cdap log file that contains a thread dump at the end.