...

Services

Add

...

Ranger Cluster Setup

What cluster to build

1.Cdap single node (must)

For Services add

Spark-historyserver

Hive-server2

Generic-users

500 200 GB

2x large

2.A separate ranger-server on any system with centOS6 (must)

create a 4.2 cdap-single node

500GB

git checkout tag tags/release-ranger-0.7.1

git checkout tags/release-ranger-0.7.1

How to install ranger on a cluster:

How to install maven on the cluster to build

Step by step

...

2 (don't do)

install ranger 0.6.3 instead

First use root

sudo su -

Install Maven

...

Install gcc

yum install gcc

Clone the ranger source code (must)

mkdir ~/dev
cd ~/dev
git clone https://github.com/apache/incubator-ranger.git
cd incubator-ranger

...

git checkout tags/release-0.6.3

Build the source (Must)

cd ~/dev/incubator-ranger

...

ranger-0.5.0-admin.tar.gz ranger-0.5.0-kms.tar.gz ranger-0.5.0-storm-plugin.tar.gz ranger-0.5.0-hbase-plugin.tar.gz ranger-0.5.0-knox-plugin.tar.gz ranger-0.5.0-usersync.tar.gz ranger-0.5.0-hdfs-plugin.tar.gz ranger-0.5.0-migration-util.tar.gz ranger-0.5.0-yarn-plugin.tar.gz ranger-0.5.0-hive-plugin.tar.gz ranger-0.5.0-solr-plugin.tar.gz ranger-0.5.0-kafka-plugin.tar.gz ranger-0.5.0-src.tar.gz

Install Steps for Ranger Policy Admin on RHEL/CentOS (must)

cd /usr/local

tar zxvf ~/dev/incubator-ranger/target/ranger-0.6.3-admin.tar.gz

ln -s ranger-0.6.3-admin ranger-admin

cd ranger-admin

...

Add these:

```
db_root_user=root
db_root_password= <enter_your_password>
```

Install and configure Solr or SolrCloud

cd ~/dev/incubator-ranger/security-admin/contrib/solr_for_audit_setup

...

Run the ./setup.sh script as root to install Solr. This will create install notes at
> /opt/solr/ranger_audit_server/install_notes.txt

./setup.sh

Start/Stop and Common Info about Solr or SolrCloud

```
Start and Stoping Solr:
Login as user solr or root and the run the below commands to start or stop Solr:

...

# * audit_solr_url URL to Solr. E.g. http://<solr_host>:6083/solr/ranger_audits
audit_solr_urls=http://localhosthostname:6083/solr/ranger_audits

...

XAAUDIT.SOLR.ENABLE=true
XAAUDIT.SOLR.URL=http://localhosthostname:6083/solr/ranger_audits

...

# DB UserId used for the XASecure schema
db_name=ranger
db_user=rangeradmin
db_password=rangeradmin
 
# DB UserId for storing auditlog infromation
audit_db_name=ranger
audit_db_user=rangerlogger
audit_db_password=rangerlogger

How to Upgrade MySQL 5.1 to MySQL 5.5 on CentOS 6.7 (Optional, only if you install ranger latest versions after 0.7)

https://www.zerostopbits.com/how-to-upgrade-mysql-5-1-to-mysql-5-5-on-centos-6-7/

1. Confirm your version of MySQL is CentOS 6.7 default

Type in the following to confirm that you have CentOS 6.7 default of MySQL 5.1 installed:

rpm -qa | grep mysql

Outputshouldlooksimilar to the following:

mysql-server-5.1.73-5.el6_6.x86_64
mysql-libs-5.1.73-5.el6_6.x86_64
mysql-5.1.73-5.el6_6.x86_64

Everything looks good, moving on!

2. Install and activate the REMI and EPEL RPM Repositories

If you have not done so already, install and activate the REMI and EPEL repositories;

wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-6.noarch.rpm && rpm -Uvh epel-release-latest-6.noarch.rpm

wget http://rpms.famillecollet.com/enterprise/remi-release-6.rpm && rpm -Uvh remi-release-6*.rpm

Now to enable the REMI repository globally:

nano /etc/yum.repos.d/remi.repo

Under the section that looks like [remi] make the following changes:

[remi]
name=Remi's RPM repository for Enterprise Linux 6 - $basearch
#baseurl=http://rpms.remirepo.net/enterprise/6/remi/$basearch/
mirrorlist=http://rpms.remirepo.net/enterprise/6/remi/mirror
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-remi

Type CTRL-O to save, and CTRL-X to exit

2. Update MySQL from 5.1 to 5.5

Simply type in the following:

yum -y update mysql*

Once that’s done, we can verify:

rpm -qa | grep mysql

And we should see something similar to the below:

mysql-5.5.45-1.el6.remi.x86_64
compat-mysql51-5.1.54-1.el6.remi.x86_64
mysql-libs-5.5.45-1.el6.remi.x86_64
mysql-server-5.5.45-1.el6.remi.x86_64

Now we need to make sure MySQL is working correctly.

If you’ve set a root password already, type the following;

mysql -u root -p

If you have not yet set a root password, you can simply type;

mysql

You should see something similar to the following

Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 1

mysql -u root -p

SET @@global.innodb_large_prefix = 1;

use

mysql_upgrade -u root -p

to solve

SQLException : SQL state: HY000 java.sql.SQLException: Column count of mysql.user is wrong. Expected 42, found 39. Created with MySQL 50173, now running 50556. Please use mysql_upgrade to fix this error. ErrorCode: 1558

error

e. Once all the required properties are updated run the setup.sh script

./setup.sh

Encountering the following error:

Will encountering the following error, if one user ranger >0.7.0 version and with a sql version <5.5:

then run

SET @@global.innodb_large_prefix = 1;

Error executing: CREATE TABLE `x_portal_user` ( `id` bigint(20) NOT NULL AUTO_INCREMENT, `create_time` datetime DEFAULT NULL, `update_time` datetime DEFAULT NULL, `added_by_id` bigint(20) DEFAULT NULL, `upd_by_id` bigint(20) DEFAULT NULL, `first_name` varchar(1022) DEFAULT NULL, `last_name` varchar(1022) DEFAULT NULL, `pub_scr_name` varchar(2048) DEFAULT NULL, `login_id` varchar(767) DEFAULT NULL, `password` varchar(512) NOT NULL, `email` varchar(512) DEFAULT NULL, `status` int(11) NOT NULL DEFAULT '0', `user_src` int(11) NOT NULL DEFAULT '0', `notes` varchar(4000) DEFAULT NULL, PRIMARY KEY (`id`), UNIQUE KEY `x_portal_user_UK_login_id` (`login_id`), UNIQUE KEY `x_portal_user_UK_email` (`email`), KEY `x_portal_user_FK_added_by_id` (`added_by_id`), KEY `x_portal_user_FK_upd_by_id` (`upd_by_id`), KEY `x_portal_user_cr_time` (`create_time`), KEY `x_portal_user_up_time` (`update_time`), KEY `x_portal_user_name` (`first_name`(767)), KEY `x_portal_user_email` (`email`), CONSTRAINT `x_portal_user_FK_added_by_id` FOREIGN KEY (`added_by_id`) REFERENCES `x_portal_user` (`id`), CONSTRAINT `x_portal_user_FK_upd_by_id` FOREIGN KEY (`upd_by_id`) REFERENCES `x_portal_user` (`id`) )AUTO_INCREMENT=2 ROW_FORMAT=DYNAMIC;
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Specified key was too long; max key length is 1000 bytes
SQLException : SQL state: 42000 com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Specified key was too long; max key length is 1000 bytes ErrorCode: 1071
2017-06-17 01:02:03,297 [E] xa_core_db.sql file import failed!
2017-06-17 01:02:03,297 [I] CORE_DB_SCHEMA might being imported by some other process
2017-06-17 01:04:03,382 [JISQL] /usr/lib/jvm/java/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/local/ranger-0.7.2-SNAPSHOT-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://localhost/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "show tables like 'xa_access_audit';"
2017-06-17 01:04:03,673 [I] Table xa_access_audit does not exist in database ranger
2017-06-17 01:04:03,673 [JISQL] /usr/lib/jvm/java/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/local/ranger-0.7.2-SNAPSHOT-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://localhost/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "delete from x_db_version_h where version='CORE_DB_SCHEMA' and active='N' and updated_by='ranger24355-1000.dev.continuuity.net';"
2017-06-17 01:04:03,955 [E] CORE_DB_SCHEMA import failed!

ranger-admin start

How to configure Ranger to work for Hbase or Hive or HDFS

How to configure Ranger and CDAP to work together.

cdap-security-extn

git checkout -b ranger-integration

...

https://www.if-not-true-then-false.com/2010/install-mysql-on-fedora-centos-red-hat-rhel/

Installing the Ranger UserSync Process (must)

mkdir /data/logs/ranger-usersync

ln -s /data/logs/ranger-usersync ranger-usersync

sudo mkdir -p /var/log/ranger-usersync

sudo chown ranger /var/log/ranger-usersync

sudo chgrp ranger /var/log/ranger-usersync

cd /usr/local

tar zxvf /root/dev/incubator-ranger/target/ranger-0.6.3-usersync.tar.gz

sudo ln -s ranger-0.6.3-usersync ranger-usersync

inside /usr/local cd ranger-usersync

POLICY_MGR_URL = http://hostname:6080

SYNC_SOURCE = unix

logdir = /var/log/ranger/usersync

export JAVA_HOME=/usr/lib/jvm/java

cd /usr/local/ranger-0.6.3-usersync

./setup.sh

./ranger-usersync-services.sh start

make sure this script run without error and there is no symlink loop.

Enabling Ranger HDFS Plugins (Optional, from HDFS, HIVE etc choose one plugin) (must choose one, suggest hive)

cd /usr/local

tar zxvf ~/dev/incubator-ranger/target/ranger-0.6.3-hdfs-plugin.tar.gz

sudo ln -s

ln -s ranger-0.6.3-hdfsplugin ranger-hdfs-plugin

cd ranger-hdfs-plugin

cd /usr/local/ranger-0.6.3-hdfs-plugin

./enable-hdfs-plugin.sh

Restart all hadoop services in:

/etc/init.d/hadoop-hdfs-datanode restart

/etc/init.d/hadoop-hdfs-namenode restart

ln -s /usr/hdp/2.3.4.7-4/hadoop/ /usr/local/hadoop

How to configure Ranger to work for Hbase or Hive or HDFS

How to configure Ranger and CDAP to work together.

Install HIVE:

/usr/hdp/2.3.4.7-4/hadoop

dbuserpassword

curl -u admin:admin -X POST -H "Accept: application/json" -H "Content-Type: application/json" -d @ranger-servicedef-cdap.json http://ranger063v24563-1000.dev.continuuity.net:6080/service/plugins/definitions

cp /home/chaoran/binding-0.1.0-SNAPSHOT.jar /usr/local/cdap-ranger/

cp /home/chaoran/lookup-0.1.0-SNAPSHOT.jar /usr/local/cdap-ranger/

cd /usr/local/cdap-ranger/

ls -l

cd /usr/local/ranger-admin/ews/webapp/WEB-INF/classes/ranger-plugins

hostname -f

cd /usr/local/ranger-admin/ews/webapp/WEB-INF/classes/ranger-plugins

mkdir cdap

chown ranger:ranger cdap

cd cdap

mv /usr/local/cdap-ranger/lookup-0.1.0-SNAPSHOT.jar ./

chown ranger:ranger lookup-0.1.0-SNAPSHOT.jar

less /var/log/cdap/master-cdap-ranger063v24563-1000.dev.continuuity.net.log

'ls /etc/init.d/ | grep cdap' ; do sudo service $i restart ; done

for i in `ls /etc/init.d/ | grep cdap` ; do sudo service $i restart ; done

tail -f /var/log/cdap/master-cdap-ranger063v24563-1000.dev.continuuity.net.log

less /var/log/cdap/master-cdap-ranger063v24563-1000.dev.continuuity.net.log

ranger-admin restart

Process for a ranger plugin code change (must)

0. Build code: mvn clean package -DskipTests

1. Delete CDAP in ranger service in panel->Access Manager

2. Delete Ranger Panel

curl -v -u admin:admin -X DELETE http://hostname:6080/service/public/v2/api/servicedef/name/cdap

3. clone a ranger repo from security extension, ranger-integration branch

build it using mvn clean package -DskipTests

4.

scp /Users/chaoranwang/Desktop/cdap-security-extn/cdap-ranger/lookup/target/lookup-0.1.0-SNAPSHOT-jar-with-dependencies.jar rangerhostname:~

ssh into rangerhostname

then cp lookup jars to, if not there use mkdir -p to make path:

/usr/local/ranger-admin/ews/webapp/WEB-INF/classes/ranger-plugins/cdap

grant

cp /home/chaoran/lookup-0.1.0-SNAPSHOT-jar-with-dependencies.jar /usr/local/ranger-admin/ews/webapp/WEB-INF/classes/ranger-plugins

chown ranger:ranger to this jar file

chown ranger:ranger cdap

cd cdap

mv /usr/local/cdap-ranger/lookup-0.1.0-SNAPSHOT.jar ./

chown ranger:ranger /usr/local/ranger-admin/ews/webapp/WEB-INF/classes/ranger-plugins/lookup-0.1.0-SNAPSHOT-jar-with-dependencies.jar

ls -l to look at user/group property to confirm it has ranger:ranger

restart ranger admin: ranger-admin restart

scp /Users/chaoranwang/Desktop/cdap-security-extn/cdap-ranger/binding/target/binding-0.1.0-SNAPSHOT.jar cdaphostname:~

mkdir -p /usr/local/cdap-ranger/

cp /home/chaoran/binding-0.1.0-SNAPSHOT.jar /usr/local/cdap-ranger/

chown cdap:cdap /usr/local/cdap-ranger/binding-0.1.0-SNAPSHOT.jar

and then restart server

restart cdap service:

for i in `ls /etc/init.d/ | grep cdap` ; do sudo service $i restart ; done

5. Create Ranger Service

curl -u admin:admin -X POST -H “Accept: application/json” -H “Content-Type: application/json” –d @ranger-servicedef-cdap.json http://hostname:6080/service/plugins/definitions

6.Debug:

cd /usr/local/ranger-admin/ on cluster

less logfile

less /usr/local/ranger-admin/ews/logs

in ranger admin look at logfile to see log location on server

logs should be generated to:

tail -f /usr/local/ranger-admin/ews/logs/catalina.out

correctly setup should show:

Image Added

Ranger plugin and CDAP architecture

Image Added

Make enforcement working

configure cdap property: first 3

https://github.com/caskdata/cdap-security-extn/tree/develop/cdap-sentry/cdap-sentry-extension

Versions Compared

Old Version 30

New Version Current

Key

Ranger Cluster Setup

What cluster to build

1.Cdap single node (must)

2.A separate ranger-server on any system with centOS6 (must)

First use root

Install Maven

Install gcc

Clone the ranger source code (must)

Build the source (Must)

Install Steps for Ranger Policy Admin on RHEL/CentOS (must)

Install and configure Solr or SolrCloud

Start/Stop and Common Info about Solr or SolrCloud

How to Upgrade MySQL 5.1 to MySQL 5.5 on CentOS 6.7 (Optional, only if you install ranger latest versions after 0.7)

Installing the Ranger UserSync Process (must)

Enabling Ranger HDFS Plugins (Optional, from HDFS, HIVE etc choose one plugin) (must choose one, suggest hive)

Restart all hadoop services in:

Install HIVE:

Process for a ranger plugin code change (must)

Ranger plugin and CDAP architecture

Make enforcement working

Page Comparison

Versions Compared

Old Version 30

New Version Current

Key

<span class="diff-html-changed" data-a11y-before="Start of changed content" data-a11y-after="End of changed content" id="changed-diff-0">[data-colorid=</span>

<span class="diff-html-added" data-a11y-before="Start of added content" data-a11y-after="End of added content" id="added-diff-0">ea881zil7b</span><span class="diff-html-changed" data-a11y-before="Start of changed content" data-a11y-after="End of changed content" id="changed-diff-1">]{color:</span>

<span class="diff-html-added" data-a11y-before="Start of added content" data-a11y-after="End of added content" id="added-diff-2">ea881zil7b</span><span class="diff-html-changed" data-a11y-before="Start of changed content" data-a11y-after="End of changed content" id="changed-diff-3">]{color:</span>

<span class="diff-html-added" data-a11y-before="Start of added content" data-a11y-after="End of added content" id="added-diff-6">w08p54dp9i</span><span class="diff-html-changed" data-a11y-before="Start of changed content" data-a11y-after="End of changed content" id="changed-diff-7">]{color:</span>

<span class="diff-html-added" data-a11y-before="Start of added content" data-a11y-after="End of added content" id="added-diff-8">w08p54dp9i</span><span class="diff-html-changed" data-a11y-before="Start of changed content" data-a11y-after="End of changed content" id="changed-diff-9">]{color:</span>

Ranger Cluster Setup

What cluster to build

1.Cdap single node (must)

2.A separate ranger-server on any system with centOS6 (must)

First use root

Install Maven

Install gcc

Clone the ranger source code (must)

Build the source (Must)

Install Steps for Ranger Policy Admin on RHEL/CentOS (must)

Install and configure Solr or SolrCloud

Start/Stop and Common Info about Solr or SolrCloud

How to Upgrade MySQL 5.1 to MySQL 5.5 on CentOS 6.7 (Optional, only if you install ranger latest versions after 0.7)

Installing the Ranger UserSync Process (must)

Enabling Ranger HDFS Plugins (Optional, from HDFS, HIVE etc choose one plugin) (must choose one, suggest hive)

Restart all hadoop services in:

Install HIVE:

Process for a ranger plugin code change (must)

Ranger plugin and CDAP architecture

Make enforcement working