Ranger Cluster Setup

What cluster to build

1.Cdap single node (must)

For Services add

Spark-historyserver

Hive-server2

Generic-users

500 200 GB

2x large

2.A separate ranger-server on any system with centOS6 (must)

create a 4.2 cdap-single node

500GB

git checkout tags/release-ranger-0.7.2 (don't do)

...

Install gcc

yum install gcc

Clone the ranger source code (must)

mkdir ~/dev
cd ~/dev
git clone https://github.com/apache/incubator-ranger.git
cd incubator-ranger

...

git checkout tags/release-0.6.3

Build the source (Must)

cd ~/dev/incubator-ranger

...

ranger-0.5.0-admin.tar.gz ranger-0.5.0-kms.tar.gz ranger-0.5.0-storm-plugin.tar.gz ranger-0.5.0-hbase-plugin.tar.gz ranger-0.5.0-knox-plugin.tar.gz ranger-0.5.0-usersync.tar.gz ranger-0.5.0-hdfs-plugin.tar.gz ranger-0.5.0-migration-util.tar.gz ranger-0.5.0-yarn-plugin.tar.gz ranger-0.5.0-hive-plugin.tar.gz ranger-0.5.0-solr-plugin.tar.gz ranger-0.5.0-kafka-plugin.tar.gz ranger-0.5.0-src.tar.gz

Install Steps for Ranger Policy Admin on RHEL/CentOS (must)

cd /usr/local

tar zxvf ~/dev/incubator-ranger/target/ranger-0.6.3-admin.tar.gz

ln -s ranger-0.6.3-admin ranger-admin

cd ranger-admin

...

# * audit_solr_url URL to Solr. E.g. http://<solr_host>:6083/solr/ranger_audits
audit_solr_urls=http://localhosthostname:6083/solr/ranger_audits

...

XAAUDIT.SOLR.ENABLE=true
XAAUDIT.SOLR.URL=http://localhosthostname:6083/solr/ranger_audits

...

# DB UserId used for the XASecure schema
db_name=ranger
db_user=rangeradmin
db_password=rangeradmin
 
# DB UserId for storing auditlog infromation
audit_db_name=ranger
audit_db_user=rangerlogger
audit_db_password=rangerlogger

How to Upgrade MySQL 5.1 to MySQL 5.5 on CentOS 6.7 (Optional, only if you install ranger latest versions after 0.7)

https://www.zerostopbits.com/how-to-upgrade-mysql-5-1-to-mysql-5-5-on-centos-6-7/

1. Confirm your version of MySQL is CentOS 6.7 default

Type in the following to confirm that you have CentOS 6.7 default of MySQL 5.1 installed:

rpm -qa | grep mysql

Outputshouldlooksimilar to the following:

mysql-server-5.1.73-5.el6_6.x86_64
mysql-libs-5.1.73-5.el6_6.x86_64
mysql-5.1.73-5.el6_6.x86_64

Everything looks good, moving on!

2. Install and activate the REMI and EPEL RPM Repositories

If you have not done so already, install and activate the REMI and EPEL repositories;

wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-6.noarch.rpm && rpm -Uvh epel-release-latest-6.noarch.rpm

wget http://rpms.famillecollet.com/enterprise/remi-release-6.rpm && rpm -Uvh remi-release-6*.rpm

Now to enable the REMI repository globally:

nano /etc/yum.repos.d/remi.repo

Under the section that looks like [remi] make the following changes:

[remi]
name=Remi's RPM repository for Enterprise Linux 6 - $basearch
#baseurl=http://rpms.remirepo.net/enterprise/6/remi/$basearch/
mirrorlist=http://rpms.remirepo.net/enterprise/6/remi/mirror
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-remi

Type CTRL-O to save, and CTRL-X to exit

2. Update MySQL from 5.1 to 5.5

Simply type in the following:

yum -y update mysql*

Once that’s done, we can verify:

rpm -qa | grep mysql

And we should see something similar to the below:

mysql-5.5.45-1.el6.remi.x86_64
compat-mysql51-5.1.54-1.el6.remi.x86_64
mysql-libs-5.5.45-1.el6.remi.x86_64
mysql-server-5.5.45-1.el6.remi.x86_64

Now we need to make sure MySQL is working correctly.

If you’ve set a root password already, type the following;

mysql -u root -p

If you have not yet set a root password, you can simply type;

mysql

You should see something similar to the following

Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 1

mysql -u root -p

SET @@global.innodb_large_prefix = 1;

use

mysql_upgrade -u root -p

to solve

SQLException : SQL state: HY000 java.sql.SQLException: Column count of mysql.user is wrong. Expected 42, found 39. Created with MySQL 50173, now running 50556. Please use mysql_upgrade to fix this error. ErrorCode: 1558

error

e. Once all the required properties are updated run the setup.sh script

./setup.sh

Encountering

Will encountering the following error, if one user ranger >0.7.0 version and with a sql version <5.5:

then run

SET @@global.innodb_large_prefix = 1;

Error executing: CREATE TABLE `x_portal_user` ( `id` bigint(20) NOT NULL AUTO_INCREMENT, `create_time` datetime DEFAULT NULL, `update_time` datetime DEFAULT NULL, `added_by_id` bigint(20) DEFAULT NULL, `upd_by_id` bigint(20) DEFAULT NULL, `first_name` varchar(1022) DEFAULT NULL, `last_name` varchar(1022) DEFAULT NULL, `pub_scr_name` varchar(2048) DEFAULT NULL, `login_id` varchar(767) DEFAULT NULL, `password` varchar(512) NOT NULL, `email` varchar(512) DEFAULT NULL, `status` int(11) NOT NULL DEFAULT '0', `user_src` int(11) NOT NULL DEFAULT '0', `notes` varchar(4000) DEFAULT NULL, PRIMARY KEY (`id`), UNIQUE KEY `x_portal_user_UK_login_id` (`login_id`), UNIQUE KEY `x_portal_user_UK_email` (`email`), KEY `x_portal_user_FK_added_by_id` (`added_by_id`), KEY `x_portal_user_FK_upd_by_id` (`upd_by_id`), KEY `x_portal_user_cr_time` (`create_time`), KEY `x_portal_user_up_time` (`update_time`), KEY `x_portal_user_name` (`first_name`(767)), KEY `x_portal_user_email` (`email`), CONSTRAINT `x_portal_user_FK_added_by_id` FOREIGN KEY (`added_by_id`) REFERENCES `x_portal_user` (`id`), CONSTRAINT `x_portal_user_FK_upd_by_id` FOREIGN KEY (`upd_by_id`) REFERENCES `x_portal_user` (`id`) )AUTO_INCREMENT=2 ROW_FORMAT=DYNAMIC;
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Specified key was too long; max key length is 1000 bytes
SQLException : SQL state: 42000 com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Specified key was too long; max key length is 1000 bytes ErrorCode: 1071
2017-06-17 01:02:03,297 [E] xa_core_db.sql file import failed!
2017-06-17 01:02:03,297 [I] CORE_DB_SCHEMA might being imported by some other process
2017-06-17 01:04:03,382 [JISQL] /usr/lib/jvm/java/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/local/ranger-0.7.2-SNAPSHOT-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://localhost/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "show tables like 'xa_access_audit';"
2017-06-17 01:04:03,673 [I] Table xa_access_audit does not exist in database ranger
2017-06-17 01:04:03,673 [JISQL] /usr/lib/jvm/java/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/local/ranger-0.7.2-SNAPSHOT-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://localhost/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "delete from x_db_version_h where version='CORE_DB_SCHEMA' and active='N' and updated_by='ranger24355-1000.dev.continuuity.net';"
2017-06-17 01:04:03,955 [E] CORE_DB_SCHEMA import failed!

...

sudo chgrp ranger /var/log/ranger-usersync

cd /usr/local

tar zxvf ~/root/dev/incubator-ranger/target/ranger-0.6.3-usersync.tar.gz

sudo ln -s ranger-0.6.3-usersync ranger-usersync

inside /usr/local cd ranger-usersync

POLICY_MGR_URL = http://localhosthostname:6080

SYNC_SOURCE = unix

logdir = /var/log/ranger/usersync

export JAVA_HOME=/usr/lib/jvm/java

sudo ln -s ranger-0.6.3-usersync ranger-usersync

cd /usr/local/ranger-0.6.3-usersync

...

./ranger-usersync-services.sh start

make sure this script run without error and there is no symlink loop.

Enabling Ranger HDFS Plugins (Optional, from HDFS, HIVE etc choose one plugin) (must choose one, suggest hive)

...

less /var/log/cdap/master-cdap-ranger063v24563-1000.dev.continuuity.net.log

ranger-admin restart

Process for a ranger plugin code change (must)

0. Build code: mvn clean package -DskipTests

...

3. clone a ranger repo from security extension, ranger-integration branch

build it using mvn clean package -DskipTests

4.

scp two jar files from codebase ranger path to server ranger path

scp /Users/chaoranwang/Desktop/cdap-security-extn/cdap-ranger/bindinglookup/target/bindinglookup-0.1.0-SNAPSHOT-jar-with-dependencies.jar hostnamerangerhostname:~/scp /Users/chaoranwang/Desktop/cdap-security-extn/cdap-ranger/lookup/target

ssh into rangerhostname

then cp lookup jars to, if not there use mkdir -p to make path:

/usr/local/ranger-admin/ews/webapp/WEB-INF/classes/ranger-plugins/cdap

grant

cp /home/chaoran/lookup-0.1.0-SNAPSHOT-jar-with-dependencies.jar hostname:~ /then cp these jars to /usr/local/cdapranger-ranger/

chown ranger:ranger to all two jar files.

ls -l to look at user/group property to confirm it has ranger:ranger

restart ranger admin: ranger-admin restart

5. Create Ranger Service

curl -u admin:admin -X POST -H “Accept: application/json” -H “Content-Type: application/json” –d @ranger-servicedef-cdap.json http://hostname:6080/service/plugins/definitions

6.Debug:

cd /usr/local/ranger-admin/ on cluster

less logfile

less /usr/local/ranger-admin/ews/logs

in ranger admin look at logfile to see log location on server

admin/ews/webapp/WEB-INF/classes/ranger-plugins

chown ranger:ranger to this jar file

chown ranger:ranger cdap

cd cdap

mv /usr/local/cdap-ranger/lookup-0.1.0-SNAPSHOT.jar ./

chown ranger:ranger /usr/local/ranger-admin/ews/webapp/WEB-INF/classes/ranger-plugins/lookup-0.1.0-SNAPSHOT-jar-with-dependencies.jar

ls -l to look at user/group property to confirm it has ranger:ranger

restart ranger admin: ranger-admin restart

scp /Users/chaoranwang/Desktop/cdap-security-extn/cdap-ranger/binding/target/binding-0.1.0-SNAPSHOT.jar cdaphostname:~

mkdir -p /usr/local/cdap-ranger/

cp /home/chaoran/binding-0.1.0-SNAPSHOT.jar /usr/local/cdap-ranger/

chown cdap:cdap /usr/local/cdap-ranger/binding-0.1.0-SNAPSHOT.jar

and then restart server

restart cdap service:

for i in `ls /etc/init.d/ | grep cdap` ; do sudo service $i restart ; done

5. Create Ranger Service

curl -u admin:admin -X POST -H “Accept: application/json” -H “Content-Type: application/json” –d @ranger-servicedef-cdap.json http://hostname:6080/service/plugins/definitions

6.Debug:

cd /usr/local/ranger-admin/ on cluster

less logfile

less /usr/local/ranger-admin/ews/logs

in ranger admin look at logfile to see log location on server

logs should be generated to:

tail -f /usr/local/ranger-admin/ews/logs/catalina.out

correctly setup should show:

Image Added

Ranger plugin and CDAP architecture

Image Added

Make enforcement working

configure cdap property: first 3

https://github.com/caskdata/cdap-security-extn/tree/develop/cdap-sentry/cdap-sentry-extension

Versions Compared

Old Version 40

New Version Current

Key

Ranger Cluster Setup

What cluster to build

1.Cdap single node (must)

2.A separate ranger-server on any system with centOS6 (must)

Install gcc

Clone the ranger source code (must)

Build the source (Must)

Install Steps for Ranger Policy Admin on RHEL/CentOS (must)

How to Upgrade MySQL 5.1 to MySQL 5.5 on CentOS 6.7 (Optional, only if you install ranger latest versions after 0.7)

Enabling Ranger HDFS Plugins (Optional, from HDFS, HIVE etc choose one plugin) (must choose one, suggest hive)

Process for a ranger plugin code change (must)

Ranger plugin and CDAP architecture

Make enforcement working

Page Comparison

Versions Compared

Old Version 40

New Version Current

Key

What cluster to build

1.Cdap single node (must)

2.A separate ranger-server on any system with centOS6 (must)

Install gcc

Clone the ranger source code (must)

Build the source (Must)

Install Steps for Ranger Policy Admin on RHEL/CentOS (must)

How to Upgrade MySQL 5.1 to MySQL 5.5 on CentOS 6.7 (Optional, only if you install ranger latest versions after 0.7)

Enabling Ranger HDFS Plugins (Optional, from HDFS, HIVE etc choose one plugin) (must choose one, suggest hive)

Process for a ranger plugin code change (must)

Ranger plugin and CDAP architecture

Make enforcement working