Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 51 Next »

Ranger Cluster Setup

What cluster to build

1.Cdap single node (must)

For Services add

Spark-historyserver

Hive-server2

Generic-users

200 GB

2x large


2.A separate ranger-server on any system with centOS6 (must)

 

 create a 4.2 cdap-single node

500GB

git checkout tags/release-ranger-0.7.2 (don't do)

install ranger 0.6.3 instead

 

First use root

sudo su -

Install Maven

 

cd /usr/local
# Download maven latest distribution tar from apache maven site
# https://maven.apache.org/download.cgi
tar -xvf apache-­maven-<Version>-­bin.tar.gz
 
export M2_HOME=/usr/local/apache-­maven-­<Version>
export M2=$M2_HOME/bin
export PATH=$M2:$PATH
 
#Now to test your install of Maven, enter the following command
mvn -version

export MAVEN_OPTS="-Xmx2048m -XX:MaxPermSize=512m"

Install git

yum install git

Install gcc

yum install gcc 

Clone the ranger source code (must)

mkdir ~/dev
cd ~/dev
git clone https://github.com/apache/incubator-ranger.git
cd incubator-ranger
https://github.com/apache/ranger/releases
check releases and choose 0.6.3
git checkout tags/release-0.6.3

Build the source (Must)

cd ~/dev/incubator-ranger

export MAVEN_OPTS="-Xmx512M" 
export JAVA_HOME=<Java Installation Directory>
#e.g. export JAVA_HOME=/usr/lib/jvm/java
#Make sure your java version is 1.7.x
$JAVA_HOME/bin/java -version
   java version "1.7.0_79"
   OpenJDK Runtime Environment (rhel-2.5.5.3.el6_6-x86_64 u79-b14)
   OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode)
export PATH=$JAVA_HOME/bin:$PATH
mvn clean compile package assembly:assembly install -DskipTests (very important, the range 0.5 installation guide omit the -DskipTests)

may encounter the out of memory error, set MAVEN_OPTS to a larger memory
ls target/*.tar.gz
   ranger-0.5.0-admin.tar.gz ranger-0.5.0-kms.tar.gz ranger-0.5.0-storm-plugin.tar.gz ranger-0.5.0-hbase-plugin.tar.gz ranger-0.5.0-knox-plugin.tar.gz ranger-0.5.0-usersync.tar.gz ranger-0.5.0-hdfs-plugin.tar.gz ranger-0.5.0-migration-util.tar.gz ranger-0.5.0-yarn-plugin.tar.gz ranger-0.5.0-hive-plugin.tar.gz ranger-0.5.0-solr-plugin.tar.gz ranger-0.5.0-kafka-plugin.tar.gz ranger-0.5.0-src.tar.gz

Install Steps for Ranger Policy Admin on RHEL/CentOS (must)

cd /usr/local

tar zxvf ~/dev/incubator-ranger/target/ranger-0.6.3-admin.tar.gz

ln -s ranger-0.6.3-admin ranger-admin

cd ranger-­admin

 

 

sudo vim install.properties

Add these:

```
db_root_user=root
db_root_password= <enter_your_password>
```

Install and configure Solr or SolrCloud

cd ~/dev/incubator-ranger/security-admin/contrib/solr_for_audit_setup

 

Edit following in install.properties

```
MAX_AUDIT_RETENTION_DAYS=2
SOLR_INSTALL=true
SOLR_DOWNLOAD_URL=http://archive.apache.org/dist/lucene/solr/5.2.1/solr-5.2.1.tgz
```

Since solr logs and data will take lots of space it important to symlink
> /opt/solr
> /var/log/solr/ranger_audits

to /data on coopr cluster

ln -s /opt/solr/ /data
ln -s /var/log/solr/ranger_audits /data

Run the ./setup.sh script as root to install Solr. This will create install notes at
> /opt/solr/ranger_audit_server/install_notes.txt

./setup.sh

 

Start/Stop and Common Info about Solr or SolrCloud

```
Start and Stoping Solr:
Login as user solr or root and the run the below commands to start or stop Solr:

To start Solr run: /opt/solr/ranger_audit_server/scripts/start_solr.sh
To stop Solr run: /opt/solr/ranger_audit_server/scripts/stop_solr.sh

After starting Solr for RangerAudit, Solr will listen at 6083. 

Configure Ranger to use the following URL   http://clustername:6083/solr/ranger_audits

Solr HOME for Ranger Audit is /opt/solr/ranger_audit_server

DATA FOLDER: /opt/solr/ranger_audit_server/data

Make sure you have enough disk space for index. In production, it is recommended to have at least 1TB free.

df -H

Filesystem Size Used Avail Use% Mounted on
/dev/sda1 493G 9.7G 458G 3% /
```

we can see solr page from here:

http://clustername:6083/solr/#/

 

For Ranger Admin, configure the following properties in install.properties:

cd /usr/local/ranger-admin

#Source for Audit DB
# * audit_db is solr or db
audit_store=solr

# * audit_solr_url URL to Solr. E.g. http://<solr_host>:6083/solr/ranger_audits
audit_solr_urls=http://localhost:6083/solr/ranger_audits

 

For all plugins, configure the following properties in install.properties

XAAUDIT.SOLR.ENABLE=true
XAAUDIT.SOLR.URL=http://localhost:6083/solr/ranger_audits

 

2. Update the install.properties 

2.1 Configure database properties. For MySQL, you will need to use the root password that you had picked while installing mysql.

 

db_root_user=root
db_root_password=<enter_your_password>
db_host=localhost

 

c.The install process would create a couple of users in the database for storing administration and audit information, pick passwords for those too. With my choices here’s how the relevant sections in the install.properties file looks like.

 

# DB UserId used for the XASecure schema
db_name=ranger
db_user=rangeradmin
db_password=rangeradmin
 
# DB UserId for storing auditlog infromation
audit_db_name=ranger
audit_db_user=rangerlogger
audit_db_password=rangerlogger

 

 

 

How to Upgrade MySQL 5.1 to MySQL 5.5 on CentOS 6.7 (Optional, only if you install ranger latest versions after 0.7)

https://www.zerostopbits.com/how-to-upgrade-mysql-5-1-to-mysql-5-5-on-centos-6-7/

1. Confirm your version of MySQL is CentOS 6.7 default

 

Type in the following to confirm that you have CentOS 6.7 default of MySQL 5.1 installed:

 

rpm -qa | grep mysql

 

Outputshouldlooksimilar to the following:

 

mysql-server-5.1.73-5.el6_6.x86_64
mysql-libs-5.1.73-5.el6_6.x86_64
mysql-5.1.73-5.el6_6.x86_64

 

Everything looks good, moving on!

 

 

2. Install and activate the REMI and EPEL RPM Repositories

 

If you have not done so already, install and activate the REMI and EPEL repositories;

 

wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-6.noarch.rpm && rpm -Uvh epel-release-latest-6.noarch.rpm

wget http://rpms.famillecollet.com/enterprise/remi-release-6.rpm && rpm -Uvh remi-release-6*.rpm

 

Now to enable the REMI repository globally:

 

nano /etc/yum.repos.d/remi.repo

 

Under the section that looks like [remi] make the following changes:

 

[remi]
name=Remi's RPM repository for Enterprise Linux 6 - $basearch
#baseurl=http://rpms.remirepo.net/enterprise/6/remi/$basearch/
mirrorlist=http://rpms.remirepo.net/enterprise/6/remi/mirror
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-remi

 

Type CTRL-O to save, and CTRL-X to exit

 

 

2. Update MySQL from 5.1 to 5.5

 

Simply type in the following:

 

yum -y update mysql*

 

Once that’s done, we can verify:

 

rpm -qa | grep mysql

 

And we should see something similar to the below:

 

mysql-5.5.45-1.el6.remi.x86_64
compat-mysql51-5.1.54-1.el6.remi.x86_64
mysql-libs-5.5.45-1.el6.remi.x86_64
mysql-server-5.5.45-1.el6.remi.x86_64

 

Now we need to make sure MySQL is working correctly.

 

If you’ve set a root password already, type the following;

 

mysql -u root -p

 

If you have not yet set a root password, you can simply type;

 

mysql

 

You should see something similar to the following

 

Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 1

 

mysql -u root -p

SET @@global.innodb_large_prefix = 1;

 

 

use 

 

mysql_upgrade -u root -p

 

to solve

 


 

SQLException : SQL state: HY000 java.sql.SQLException: Column count of mysql.user is wrong. Expected 42, found 39. Created with MySQL 50173, now running 50556. Please use mysql_upgrade to fix this error. ErrorCode: 1558

 

error

 

e. Once all the required properties are updated run the setup.sh script

 

./setup.sh

 

 

 

Will encountering the following error, if one user ranger >0.7.0 version and with a sql version <5.5:

then run


SET @@global.innodb_large_prefix = 1;

 

Error executing: CREATE TABLE `x_portal_user` ( `id` bigint(20) NOT NULL AUTO_INCREMENT, `create_time` datetime DEFAULT NULL, `update_time` datetime DEFAULT NULL, `added_by_id` bigint(20) DEFAULT NULL, `upd_by_id` bigint(20) DEFAULT NULL, `first_name` varchar(1022) DEFAULT NULL, `last_name` varchar(1022) DEFAULT NULL, `pub_scr_name` varchar(2048) DEFAULT NULL, `login_id` varchar(767) DEFAULT NULL, `password` varchar(512) NOT NULL, `email` varchar(512) DEFAULT NULL, `status` int(11) NOT NULL DEFAULT '0', `user_src` int(11) NOT NULL DEFAULT '0', `notes` varchar(4000) DEFAULT NULL, PRIMARY KEY (`id`), UNIQUE KEY `x_portal_user_UK_login_id` (`login_id`), UNIQUE KEY `x_portal_user_UK_email` (`email`), KEY `x_portal_user_FK_added_by_id` (`added_by_id`), KEY `x_portal_user_FK_upd_by_id` (`upd_by_id`), KEY `x_portal_user_cr_time` (`create_time`), KEY `x_portal_user_up_time` (`update_time`), KEY `x_portal_user_name` (`first_name`(767)), KEY `x_portal_user_email` (`email`), CONSTRAINT `x_portal_user_FK_added_by_id` FOREIGN KEY (`added_by_id`) REFERENCES `x_portal_user` (`id`), CONSTRAINT `x_portal_user_FK_upd_by_id` FOREIGN KEY (`upd_by_id`) REFERENCES `x_portal_user` (`id`) )AUTO_INCREMENT=2 ROW_FORMAT=DYNAMIC;
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Specified key was too long; max key length is 1000 bytes
SQLException : SQL state: 42000 com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Specified key was too long; max key length is 1000 bytes ErrorCode: 1071
2017-06-17 01:02:03,297 [E] xa_core_db.sql file import failed!
2017-06-17 01:02:03,297 [I] CORE_DB_SCHEMA might being imported by some other process
2017-06-17 01:04:03,382 [JISQL] /usr/lib/jvm/java/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/local/ranger-0.7.2-SNAPSHOT-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://localhost/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "show tables like 'xa_access_audit';"
2017-06-17 01:04:03,673 [I] Table xa_access_audit does not exist in database ranger
2017-06-17 01:04:03,673 [JISQL] /usr/lib/jvm/java/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/local/ranger-0.7.2-SNAPSHOT-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://localhost/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "delete from x_db_version_h where version='CORE_DB_SCHEMA' and active='N' and updated_by='ranger24355-1000.dev.continuuity.net';"
2017-06-17 01:04:03,955 [E] CORE_DB_SCHEMA import failed!

ranger-admin start

 

 

 

 

 

 

 

cdap-security-extn

git checkout -b ranger-integration



Install mysql 5.7 on cent 6

https://www.if-not-true-then-false.com/2010/install-mysql-on-fedora-centos-red-hat-rhel/

 

 

 

Installing the Ranger UserSync Process (must)

mkdir /data/logs/ranger-usersync

ln -s /data/logs/ranger-usersync ranger-usersync

sudo mkdir -p /var/log/ranger-usersync

sudo chown ranger /var/log/ranger-usersync

sudo chgrp ranger /var/log/ranger-usersync

cd /usr/local

tar zxvf /root/dev/incubator-ranger/target/ranger-0.6.3-usersync.tar.gz

sudo ln -s ranger-0.6.3-usersync ranger-usersync

inside /usr/local cd ranger-usersync


POLICY_MGR_URL = http://localhost:6080
SYNC_SOURCE = unix 
logdir = /var/log/ranger/usersync

export JAVA_HOME=/usr/lib/jvm/java

 

cd /usr/local/ranger-0.6.3-usersync

./setup.sh

./ranger-usersync-services.sh start

 

Enabling Ranger HDFS Plugins (Optional, from HDFS, HIVE etc choose one plugin) (must choose one, suggest hive)

cd /usr/local

tar zxvf ~/dev/incubator-ranger/target/ranger-0.6.3-hdfs-plugin.tar.gz

sudo ln -­s 

ln -s ranger-0.6.3-hdfs­plugin ranger-­hdfs-­plugin

cd ranger-­hdfs-­plugin

 

cd /usr/local/ranger-0.6.3-hdfs-plugin

./enable-­hdfs-­plugin.sh

 

Restart all hadoop services in:

/etc/init.d/hadoop-hdfs-datanode restart

/etc/init.d/hadoop-hdfs-namenode restart

 

 

ln -s /usr/hdp/2.3.4.7-4/hadoop/ /usr/local/hadoop

 

 

 

How to configure Ranger to work  for Hbase or Hive or HDFS 

 

How to configure Ranger and CDAP to work together.

 

Install HIVE:


/usr/hdp/2.3.4.7-4/hadoop

  

dbuserpassword

curl -u admin:admin -X POST -H "Accept: application/json" -H "Content-Type: application/json" -d @ranger-servicedef-cdap.json http://ranger063v24563-1000.dev.continuuity.net:6080/service/plugins/definitions

 

cp /home/chaoran/binding-0.1.0-SNAPSHOT.jar /usr/local/cdap-ranger/

cp /home/chaoran/lookup-0.1.0-SNAPSHOT.jar  /usr/local/cdap-ranger/

cd /usr/local/cdap-ranger/

ls -l

cd /usr/local/ranger-admin/ews/webapp/WEB-INF/classes/ranger-plugins

hostname -f

cd /usr/local/ranger-admin/ews/webapp/WEB-INF/classes/ranger-plugins

mkdir cdap

chown ranger:ranger cdap

cd cdap

mv /usr/local/cdap-ranger/lookup-0.1.0-SNAPSHOT.jar ./

chown ranger:ranger lookup-0.1.0-SNAPSHOT.jar

less /var/log/cdap/master-cdap-ranger063v24563-1000.dev.continuuity.net.log

'ls /etc/init.d/ | grep cdap' ; do sudo service $i restart ; done

for i in `ls /etc/init.d/ | grep cdap` ; do sudo service $i restart ; done

tail -f /var/log/cdap/master-cdap-ranger063v24563-1000.dev.continuuity.net.log

less /var/log/cdap/master-cdap-ranger063v24563-1000.dev.continuuity.net.log

ranger-admin restart

 

Process for a ranger plugin code change (must)

0. Build code: mvn clean package -DskipTests

1. Delete CDAP in ranger service in panel->Access Manager

2. Delete Ranger Panel

curl -v -u admin:admin -X DELETE http://hostname:6080/service/public/v2/api/servicedef/name/cdap

3. clone a ranger repo from security extension, ranger-integration branch

build it using mvn clean package -DskipTests

4.

 

scp /Users/chaoranwang/Desktop/cdap-security-extn/cdap-ranger/lookup/target/lookup-0.1.0-SNAPSHOT-jar-with-dependencies.jar rangerhostname:~

ssh into rangerhostname

then cp lookup jars to, if not there use mkdir -p to make path

cp /home/chaoran/lookup-0.1.0-SNAPSHOT-jar-with-dependencies.jar /usr/local/ranger-admin/ews/webapp/WEB-INF/classes/ranger-plugins

chown ranger:ranger to this jar file

ls -l to look at user/group property to confirm it has ranger:ranger

restart ranger admin: ranger-admin restart

 

scp /Users/chaoranwang/Desktop/cdap-security-extn/cdap-ranger/binding/target/binding-0.1.0-SNAPSHOT.jar cdaphostname:~

cp binding jar from cdap server to /usr/local/cdap-ranger/binding-0.1.0-SNAPSHOT.jar

chown cdap:cdap to this jar file

and then restart server

restart cdap service:

for i in `ls /etc/init.d/ | grep cdap` ; do sudo service $i restart ; done

 

 

 

 

5. Create Ranger Service

curl -u admin:admin -X POST -H “Accept: application/json” -H “Content-Type: application/json” –d @ranger-servicedef-cdap.json http://hostname:6080/service/plugins/definitions

6.Debug: 

cd /usr/local/ranger-admin/ on cluster

less logfile 

less /usr/local/ranger-admin/ews/logs

in ranger admin look at logfile to see log location on server

logs should be generated to:

tail -f /usr/local/ranger-admin/ews/logs/catalina.out


Make enforcement working

configure cdap property: first 3

https://github.com/caskdata/cdap-security-extn/tree/develop/cdap-sentry/cdap-sentry-extension

 

 

 

 

 

 

 

 

 

  • No labels

Created in 2020 by Google Inc.