thehyve/puppet-data_showcase

Name: puppet-data_showcase

Owner: The Hyve

Description: Puppet module for the data showcase application

Created: 2017-09-20 18:48:31.0

Updated: 2017-09-20 19:06:53.0

Pushed: 2017-10-25 21:06:19.0

Homepage: null

Size: 37

Language: Puppet

GitHub Committers

User	Most Recent Commit	# Commits

Other Committers

User	Email	Most Recent Commit	# Commits

README

Puppet module for the Data showcase.

This is the repository containing a puppet module for deploying the Data showcase application, which can be found at https://github.com/thehyve/data-showcase.

The module creates system user datashowcase, and installs and configures the service data-showcase. The repository used to fetch the required packages is repo.thehyve.nl.

Dependencies and installation

Operating system packages

The module expects certain packages to be available through the package manager of the operating system (e.g., yum or apt):

• For all components:
• java-1.8.0-openjdk

Puppet modules

The module depends on the archive and java puppet modules.

The most convenient way is to run puppet module install as root:

 puppet module install puppet-archive
 puppet module install puppetlabs-java

If you want to let the module install PostgreSQL as well, install the postgresql module:

 puppet module install puppetlabs-postgresql

Install the data_showcase module

Copy the data_showcase module repository to the /etc/puppet/modules directory:

etc/puppet/modules
clone https://github.com/thehyve/puppet-data_showcase.git data_showcase

Configuration

The node manifest

For each node where you want to install the data showcase, the module needs to be included with include ::data_showcase::complete.

Here is an example manifest file manifests/test.example.com.pp:

 'test.example.com' {
include ::data_showcase::complete

The node manifest can also be in another file, e.g., site.pp.

Configuring a node using Hiera

It is preferred to configure the module parameters using Hiera.

To activate the use of Hiera, configure /etc/puppet/hiera.yaml. Example:

kends:
yaml
l:
atadir: '/etc/puppet/hieradata'
rarchy:
'%{::clientcert}'
'default'

Defaults can then be configured in /etc/puppet/hieradata/default.yaml, e.g.:

_showcase::environment: Public

Machine specific configuration should be in /etc/puppet/hieradata/${hostname}.yaml, e.g., /etc/puppet/hieradata/test.example.com.yaml:

_showcase::db_user: test
_showcase::db_password: my secret
_showcase::db_host: 10.0.2.2
_showcase::db_name: transmart
_showcase::db_port: 1521
_showcase::memory: 4g
_showcase::environment: Public
_showcase::version: 0.0.1-SNAPSHOT
_showcase::access_token: configure a secret token

Configuring a node in the manifest file

Alternatively, the node specific configuration can also be done with class parameters in the node manifest. Here is an example:

 'test.example.com' {
# Site specific configuration for the data showcase
class { '::data_showcase::params':
    db_user      => 'test',
    db_password  => 'my secret',
    db_port      => 1521,
    db_name      => 'transmart,
    access_token => 'configure a secret token',
}

include ::data_showcase::complete

Masterless installation

It is also possible to use the module without a Puppet master by applying a manifest directly using puppet apply.

There is an example manifest in examples/complete.pp for generating the required configuration files and installing data-showcase.

 puppet apply --modulepath=${modulepath} examples/complete.pp

where modulepath is a list of directories where Puppet can find modules in, separated by the system path-separator character (on Ubuntu/CentOS it is :). Example:

 puppet apply --modulepath=${HOME}/puppet/:/etc/puppet/modules examples/complete.pp

Database installation and preparation

Create a PostgreSQL database

To install postgresql with the database admin credentials, run:

 puppet apply --modulepath=${modulepath} examples/postgres.pp

Prepare the database for the data showcase

 -iu postgres psql

Create the database:

te user datashowcase with password 'my secret';
te database data_showcase;
t all privileges on database data_showcase to datashowcase;

Load test data

=test.json
n=configure a secret token
_showcase_url=http://localhost:8080

tch test data file
 -o "${file}" https://raw.githubusercontent.com/thehyve/data-showcase/master/data-showcase/src/main/resources/test.json

load data
 -v -F "data=@${file}" "${data_showcase_url}/api/data_import/upload?requestToken=${token}"

Manage systemd services

Start the data-showcase service:

 systemctl start data-showcase

Check the status of the service:

 systemctl status data-showcase

Stop the service:

 systemctl stop data-showcase

Check a full log of service build

nalctl -u data-showcase - build log

Test

The module has been tested on Ubuntu 16.04 with Puppet version 4. There are some automated tests, run using rake.

A version of ruby before 2.3 is required. rvm can be used to install a specific version of ruby. Use rvm install 2.1 to use ruby version 2.1.

Rake tests

Install rake using the system-wide ruby:

install ruby-devel
install bundler
rt PUPPET_VERSION=4.4.2
le

or using rvm:

install 2.1
install bundler
rt PUPPET_VERSION=4.4.2
le

Run the test suite:

 test

Classes

Overview of the classes defined in this module.

| Class name | Description | |————|————-| | ::data_showcase | Creates the system users. | | ::data_showcase::config | Generates the application configuration. | | ::data_showcase::service | Installs the data-showcase service. | | ::data_showcase::complete | Installs all of the above. | | ::data_showcase::database | Installs postgresql with the database admin credentials. |

Module parameters

Overview of the parameters that can be used in Hiera to configure the module. Alternatively, the parameters of the ::transmart_core::params class can be used to configure these settings.

| Hiera key | Default value | Description | |———–|—————|————-| | data_showcase::version | 0.0.1-SNAPSHOT | The version of the data showcase to install. | | data_showcase::nexus_url | https://repo.thehyve.nl | The Nexus/Maven repository server. | | data_showcase::repository | releases | The repository to use. [snapshots, releases] | | data_showcase::user | datashowcase | System user that runs the application. | | data_showcase::user_home | /home/${user} | The user home directory | | data_showcase::db_user | | The database admin username. (Mandatory) | | data_showcase::db_password | | The database admin password. (Mandatory) | | data_showcase::db_host | localhost | The database server host name. | | data_showcase::db_port | 5432 | The database server port. | | data_showcase::db_name | data-showcase | The database name. | | data_showcase::access_token | | Access token for loading data. (Mandatory) | | data_showcase::memory | 2g | The memory limit for the JVM. | | data_showcase::app_port | 8080 | The port the data-showcase application runs on. | | data_showcase::environment | Public | The environment to configure the system for [Public, Internal]. | | data_showcase::vu_logo_file | | File name of the logo to use. The logo should be in ${user_home}/images. | | data_showcase::ntr_logo_file | | File name of the logo to use. The logo should be in ${user_home}/images. |

Images

The example image in files is a painting by Frans Francken the Younger.

License

Copyright © 2017 The Hyve.

The puppet module for TranSMART is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see https://www.gnu.org/licenses/.