Ranger Installation

ranger cluster setup

Services

Add

Spark-historyserver

Hive-server2

Generic-users

500 GB

2x large

create a 4.2 cdap-single node

git checkout tag release-ranger-0.7.1

git checkout tags/release-ranger-0.7.1

How to install ranger on a cluster:

How to install maven on the cluster to build

Step by step

First use root

sudo su -

Install Maven

cd /usr/local

# Download maven latest distribution tar from apache maven site

# https://maven.apache.org/download.cgi

tar -xvf apache-maven-<Version>-bin.tar.gz

export M2_HOME=/usr/local/apache-maven-<Version>

export M2=$M2_HOME/bin

export PATH=$M2:$PATH

#Now to test your install of Maven, enter the following command

mvn -version

export MAVEN_OPTS="-Xmx2048m -XX:MaxPermSize=512m"

mvn clean compile package assembly:assembly install -DskipTests

Install git

yum install git

Install gcc

yum install gcc

Clone the ranger source code

mkdir ~/dev
cd ~/dev
git clone https://github.com/apache/incubator-ranger.git
cd incubator-ranger

git checkout tags/release-ranger-0.7.1

Build the source

cd ~/dev/incubator-ranger

export MAVEN_OPTS="-Xmx512M"

export JAVA_HOME=<Java Installation Directory>

#e.g. export JAVA_HOME=/usr/lib/jvm/java

#Make sure your java version is 1.7.x

$JAVA_HOME/bin/java -version

java version "1.7.0_79"

OpenJDK Runtime Environment (rhel-2.5.5.3.el6_6-x86_64 u79-b14)

OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode)

export PATH=$JAVA_HOME/bin:$PATH

sudo mvn clean compile package assembly:assembly install -DskipTests (very important, the range 0.5 installation guide omit the -DskipTests)

may encounter the out of memory error, set MAVEN_OPTS to a larger memory

ls target/*.tar.gz

ranger-0.5.0-admin.tar.gz ranger-0.5.0-kms.tar.gz ranger-0.5.0-storm-plugin.tar.gz ranger-0.5.0-hbase-plugin.tar.gz ranger-0.5.0-knox-plugin.tar.gz ranger-0.5.0-usersync.tar.gz ranger-0.5.0-hdfs-plugin.tar.gz ranger-0.5.0-migration-util.tar.gz ranger-0.5.0-yarn-plugin.tar.gz ranger-0.5.0-hive-plugin.tar.gz ranger-0.5.0-solr-plugin.tar.gz ranger-0.5.0-kafka-plugin.tar.gz ranger-0.5.0-src.tar.gz

Install Steps for Ranger Policy Admin on RHEL/CentOS

cd /usr/local

tar zxvf ~/dev/incubator-ranger/target/ranger-0.7.1-admin.tar.gz

ln -s ranger-0.7.1-admin ranger-admin

cd ranger-admin

sudo vim install.properties

Add these:

```
db_root_user=root
db_root_password= <enter_your_password>
```

Install and configure Solr or SolrCloud

cd ~/dev/incubator-ranger/security-admin/contrib/solr_for_audit_setup

Edit following in install.properties

```
MAX_AUDIT_RETENTION_DAYS=2
SOLR_INSTALL=true
SOLR_DOWNLOAD_URL=http://archive.apache.org/dist/lucene/solr/5.2.1/solr-5.2.1.tgz
```

Since solr logs and data will take lots of space it important to symlink
> /opt/solr
> /var/log/solr/ranger_audits

to /data on coopr cluster

ln -s /opt/solr/ /data
ln -s /var/log/solr/ranger_audits /data

Run the ./setup.sh script as root to install Solr. This will create install notes at
> /opt/solr/ranger_audit_server/install_notes.txt

./setup.sh

Start/Stop and Common Info about Solr or SolrCloud

```
Start and Stoping Solr:
Login as user solr or root and the run the below commands to start or stop Solr:

To start Solr run: /opt/solr/ranger_audit_server/scripts/start_solr.sh
To stop Solr run: /opt/solr/ranger_audit_server/scripts/stop_solr.sh

After starting Solr for RangerAudit, Solr will listen at 6083.

Configure Ranger to use the following URL http://clustername:6083/solr/ranger_audits

Solr HOME for Ranger Audit is /opt/solr/ranger_audit_server

DATA FOLDER: /opt/solr/ranger_audit_server/data

Make sure you have enough disk space for index. In production, it is recommended to have at least 1TB free.

df -H

Filesystem Size Used Avail Use% Mounted on
/dev/sda1 493G 9.7G 458G 3% /
```

we can see solr page from here:

http://clustername:6083/solr/#/

For Ranger Admin, configure the following properties in install.properties:

cd /usr/local/ranger-admin

#Source for Audit DB
# * audit_db is solr or db
audit_store=solr

# * audit_solr_url URL to Solr. E.g. http://<solr_host>:6083/solr/ranger_audits
audit_solr_urls=http://localhost:6083/solr/ranger_audits

For all plugins, configure the following properties in install.properties

XAAUDIT.SOLR.ENABLE=true
XAAUDIT.SOLR.URL=http://localhost:6083/solr/ranger_audits

2. Update the install.properties

2.1 Configure database properties. For MySQL, you will need to use the root password that you had picked while installing mysql.

db_root_user=root
db_root_password=<enter_your_password>
db_host=localhost

c.The install process would create a couple of users in the database for storing administration and audit information, pick passwords for those too. With my choices here’s how the relevant sections in the install.properties file looks like.

# DB UserId used for the XASecure schema
db_name=ranger
db_user=rangeradmin
db_password=rangeradmin
 
# DB UserId for storing auditlog infromation
audit_db_name=ranger
audit_db_user=rangerlogger
audit_db_password=rangerlogger

How to Upgrade MySQL 5.1 to MySQL 5.5 on CentOS 6.7

https://www.zerostopbits.com/how-to-upgrade-mysql-5-1-to-mysql-5-5-on-centos-6-7/

1. Confirm your version of MySQL is CentOS 6.7 default

Type in the following to confirm that you have CentOS 6.7 default of MySQL 5.1 installed:

rpm -qa | grep mysql

Outputshouldlooksimilar to the following:

mysql-server-5.1.73-5.el6_6.x86_64
mysql-libs-5.1.73-5.el6_6.x86_64
mysql-5.1.73-5.el6_6.x86_64

Everything looks good, moving on!

2. Install and activate the REMI and EPEL RPM Repositories

If you have not done so already, install and activate the REMI and EPEL repositories;

wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-6.noarch.rpm && rpm -Uvh epel-release-latest-6.noarch.rpm

wget http://rpms.famillecollet.com/enterprise/remi-release-6.rpm && rpm -Uvh remi-release-6*.rpm

Now to enable the REMI repository globally:

nano /etc/yum.repos.d/remi.repo

Under the section that looks like [remi] make the following changes:

[remi]
name=Remi's RPM repository for Enterprise Linux 6 - $basearch
#baseurl=http://rpms.remirepo.net/enterprise/6/remi/$basearch/
mirrorlist=http://rpms.remirepo.net/enterprise/6/remi/mirror
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-remi

Type CTRL-O to save, and CTRL-X to exit

2. Update MySQL from 5.1 to 5.5

Simply type in the following:

yum -y update mysql*

Once that’s done, we can verify:

rpm -qa | grep mysql

And we should see something similar to the below:

mysql-5.5.45-1.el6.remi.x86_64
compat-mysql51-5.1.54-1.el6.remi.x86_64
mysql-libs-5.5.45-1.el6.remi.x86_64
mysql-server-5.5.45-1.el6.remi.x86_64

Now we need to make sure MySQL is working correctly.

If you’ve set a root password already, type the following;

mysql -u root -p

If you have not yet set a root password, you can simply type;

mysql

You should see something similar to the following

Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 1

mysql -u root -p

SET @@global.innodb_large_prefix = 1;

use

mysql_upgrade -u root -p

to solve

SQLException : SQL state: HY000 java.sql.SQLException: Column count of mysql.user is wrong. Expected 42, found 39. Created with MySQL 50173, now running 50556. Please use mysql_upgrade to fix this error. ErrorCode: 1558

error

e. Once all the required properties are updated run the setup.sh script

./setup.sh

Encountering the following error:

then run

SET @@global.innodb_large_prefix = 1;

Error executing: CREATE TABLE `x_portal_user` ( `id` bigint(20) NOT NULL AUTO_INCREMENT, `create_time` datetime DEFAULT NULL, `update_time` datetime DEFAULT NULL, `added_by_id` bigint(20) DEFAULT NULL, `upd_by_id` bigint(20) DEFAULT NULL, `first_name` varchar(1022) DEFAULT NULL, `last_name` varchar(1022) DEFAULT NULL, `pub_scr_name` varchar(2048) DEFAULT NULL, `login_id` varchar(767) DEFAULT NULL, `password` varchar(512) NOT NULL, `email` varchar(512) DEFAULT NULL, `status` int(11) NOT NULL DEFAULT '0', `user_src` int(11) NOT NULL DEFAULT '0', `notes` varchar(4000) DEFAULT NULL, PRIMARY KEY (`id`), UNIQUE KEY `x_portal_user_UK_login_id` (`login_id`), UNIQUE KEY `x_portal_user_UK_email` (`email`), KEY `x_portal_user_FK_added_by_id` (`added_by_id`), KEY `x_portal_user_FK_upd_by_id` (`upd_by_id`), KEY `x_portal_user_cr_time` (`create_time`), KEY `x_portal_user_up_time` (`update_time`), KEY `x_portal_user_name` (`first_name`(767)), KEY `x_portal_user_email` (`email`), CONSTRAINT `x_portal_user_FK_added_by_id` FOREIGN KEY (`added_by_id`) REFERENCES `x_portal_user` (`id`), CONSTRAINT `x_portal_user_FK_upd_by_id` FOREIGN KEY (`upd_by_id`) REFERENCES `x_portal_user` (`id`) )AUTO_INCREMENT=2 ROW_FORMAT=DYNAMIC;
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Specified key was too long; max key length is 1000 bytes
SQLException : SQL state: 42000 com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Specified key was too long; max key length is 1000 bytes ErrorCode: 1071
2017-06-17 01:02:03,297 [E] xa_core_db.sql file import failed!
2017-06-17 01:02:03,297 [I] CORE_DB_SCHEMA might being imported by some other process
2017-06-17 01:04:03,382 [JISQL] /usr/lib/jvm/java/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/local/ranger-0.7.2-SNAPSHOT-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://localhost/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "show tables like 'xa_access_audit';"
2017-06-17 01:04:03,673 [I] Table xa_access_audit does not exist in database ranger
2017-06-17 01:04:03,673 [JISQL] /usr/lib/jvm/java/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/local/ranger-0.7.2-SNAPSHOT-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://localhost/ranger -u 'rangeradmin' -p '********' -noheader -trim -c \; -query "delete from x_db_version_h where version='CORE_DB_SCHEMA' and active='N' and updated_by='ranger24355-1000.dev.continuuity.net';"
2017-06-17 01:04:03,955 [E] CORE_DB_SCHEMA import failed!

ranger-admin start

How to configure Ranger to work for Hbase or Hive or HDFS

How to configure Ranger and CDAP to work together.

cdap-security-extn

git checkout -b ranger-integration