ballerina-guides/data-backed-service

Name: data-backed-service

Owner: ballerina-guides

Description: null

Created: 2018-02-16 13:48:39.0

Updated: 2018-05-01 07:51:01.0

Pushed: 2018-05-01 07:51:01.0

Homepage: null

Size: 570

Language: Ballerina

GitHub Committers

UserMost Recent Commit# Commits

Other Committers

UserEmailMost Recent Commit# Commits

README

Build Status

Database Interaction

Data inside a database can be exposed to the outside world by using a database backed RESTful web service. RESTful API calls enable you to add, view, update, and remove data stored in a database from the outside world.

This guide walks you through building a database-backed RESTful web service with Ballerina.

The following are the sections available in this guide.

What you'll build

You'll build an employee data management REST service that performs CRUD Operations (Create, Read, Update, Delete) on the MySQL database. Also, this guide walks you through the process of accessing relational data via the Ballerina language. The service will have following functionalities.

Basically, this service will deal with a MySQL database and expose the data operations as a web service. Refer to the following diagram to understand the complete end-to-end scenario.

alt text

Prerequisites
Optional requirements
Implementation

If you want to skip the basics, you can download the git repo and directly move to “Testing” section by skipping “Developing” section.

Create the project structure

Ballerina is a complete programming language that can have any custom project structure that you wish. Although the language allows you to have any package structure, use the following package structure for this project to follow this guide.

-backed-service
 guide
 ??? data_backed_service
 |    ??? employee_db_service.bal
 |    ??? test
 |        ??? employee_db_service_test.bal
 ???ballerina.conf
Developing the SQL data backed web service

Ballerina language has built-in support for writing web services. The service keyword in Ballerina simply defines a web service. Inside the service block, we can have all the required resources. You can define a resource inside the service. You can implement the business logic inside a resource using Ballerina language syntaxes. We can use the following database schema to store employee data.

---------+-------------+------+-----+---------+-------+
eld      | Type        | Null | Key | Default | Extra |
---------+-------------+------+-----+---------+-------+
ployeeID | int(11)     | NO   | PRI | NULL    |       |
me       | varchar(50) | YES  |     | NULL    |       |
e        | int(11)     | YES  |     | NULL    |       |
N        | int(11)     | YES  |     | NULL    |       |
---------+-------------+------+-----+---------+-------+

The following Ballerina code is the employee data service with resources to add, retrieve, update and delete employee data.

rt ballerina/sql;
rt ballerina/mysql;
rt ballerina/log;
rt ballerina/http;

 Employee {
string name;
int age;
int ssn;
int employeeId;


reate SQL endpoint to MySQL database
oint mysql:Client employeeDB {
host: config:getAsString("DATABASE_HOST", default = "localhost"),
port: config:getAsInt("DATABASE_PORT", default = 3306),
name: config:getAsString("DATABASE_NAME", default = "EMPLOYEE_RECORDS"),
username: config:getAsString("DATABASE_USERNAME", default = "root"),
password: config:getAsString("DATABASE_PASSWORD", default = "root"),
dbOptions: { useSSL: false }


oint http:Listener listener {
port: 9090


ervice for the employee data service
p:ServiceConfig {
basePath: "/records"

ice<http:Service> EmployeeData bind listener {

@http:ResourceConfig {
    methods: ["POST"],
    path: "/employee/"
}
addEmployeeResource(endpoint httpConnection, http:Request request) {
    // Initialize an empty http response message
    http:Response response;
    Employee employeeData;
    // Extract the data from the request payload
    var payloadJson = check request.getJsonPayload();
    employeeData = check <Employee>payloadJson;

    // Check for errors with JSON payload using
    if (employeeData.name == "" || employeeData.age == 0 || employeeData.ssn == 0 ||
        employeeData.employeeId == 0) {
        response.setTextPayload("Error : json payload should contain
         {name:<string>, age:<int>, ssn:<123456>,employeeId:<int>} ");
        response.statusCode = 400;
        _ = httpConnection->respond(response);
        done;
    }

    // Invoke insertData function to save data in the Mymysql database
    json ret = insertData(employeeData.name, employeeData.age, employeeData.ssn,
        employeeData.employeeId);
    // Send the response back to the client with the employee data
    response.setJsonPayload(ret);
    _ = httpConnection->respond(response);
}

@http:ResourceConfig {
    methods: ["GET"],
    path: "/employee/{employeeId}"
}
retrieveEmployeeResource(endpoint httpConnection, http:Request request, string
employeeId) {
    // Initialize an empty http response message
    http:Response response;
    // Convert the employeeId string to integer
    int empID = check <int>employeeId;
    // Invoke retrieveById function to retrieve data from Mymysql database
    var employeeData = retrieveById(empID);
    // Send the response back to the client with the employee data
    response.setJsonPayload(employeeData);
    _ = httpConnection->respond(response);
}

@http:ResourceConfig {
    methods: ["PUT"],
    path: "/employee/"
}
updateEmployeeResource(endpoint httpConnection, http:Request request) {
    // Initialize an empty http response message
    http:Response response;
    Employee employeeData;
    // Extract the data from the request payload
    var payloadJson = check request.getJsonPayload();
    employeeData = check <Employee>payloadJson;

    if (employeeData.name == "" || employeeData.age == 0 || employeeData.ssn == 0 ||
        employeeData.employeeId == 0) {
        response.setTextPayload("Error : json payload should contain
         {name:<string>, age:<int>, ssn:<123456>,employeeId:<int>} ");
        response.statusCode = 400;
        _ = httpConnection->respond(response);
        done;
    }

    // Invoke updateData function to update data in mysql database
    json ret = updateData(employeeData.name, employeeData.age, employeeData.ssn,
        employeeData.employeeId);
    // Send the response back to the client with the employee data
    response.setJsonPayload(ret);
    _ = httpConnection->respond(response);
}

@http:ResourceConfig {
    methods: ["DELETE"],
    path: "/employee/{employeeId}"
}
deleteEmployeeResource(endpoint httpConnection, http:Request request, string
employeeId) {
    // Initialize an empty http response message
    http:Response response;
    // Convert the employeeId string to integer
    var empID = check <int>employeeId;
    var deleteStatus = deleteData(empID);
    // Send the response back to the client with the employee data
    response.setJsonPayload(deleteStatus);
    _ = httpConnection->respond(response);
}


ic function insertData(string name, int age, int ssn, int employeeId) returns (json){
json updateStatus;
string sqlString =
"INSERT INTO EMPLOYEES (Name, Age, SSN, EmployeeID) VALUES (?,?,?,?)";
// Insert data to SQL database by invoking update action
var ret = employeeDB->update(sqlString, name, age, ssn, employeeId);
// Use match operator to check the validity of the result from database
match ret {
    int updateRowCount => {
        updateStatus = { "Status": "Data Inserted Successfully" };
    }
    error err => {
        updateStatus = { "Status": "Data Not Inserted", "Error": err.message };
    }
}
return updateStatus;


ic function retrieveById(int employeeID) returns (json) {
json jsonReturnValue;
string sqlString = "SELECT * FROM EMPLOYEES WHERE EmployeeID = ?";
// Retrieve employee data by invoking select action defined in ballerina sql connector
var ret = employeeDB->select(sqlString, (), employeeID);
match ret {
    table dataTable => {
        // Convert the sql data table into JSON using type conversion
        jsonReturnValue = check <json>dataTable;
    }
    error err => {
        jsonReturnValue = { "Status": "Data Not Found", "Error": err.message };
    }
}
return jsonReturnValue;


ic function updateData(string name, int age, int ssn, int employeeId) returns (json){
json updateStatus = {};
string sqlString =
"UPDATE EMPLOYEES SET Name = ?, Age = ?, SSN = ? WHERE EmployeeID  = ?";
// Update existing data by invoking update action defined in ballerina sql connector
var ret = employeeDB->update(sqlString, name, age, ssn, employeeId);
match ret {
    int updateRowCount => {
        if (updateRowCount > 0) {
            updateStatus = { "Status": "Data Updated Successfully" };
        }
        else {
            updateStatus = { "Status": "Data Not Updated" };
        }
    }
    error err => {
        updateStatus = { "Status": "Data Not Updated", "Error": err.message };
    }
}
return updateStatus;


ic function deleteData(int employeeID) returns (json) {
json updateStatus = {};

string sqlString = "DELETE FROM EMPLOYEES WHERE EmployeeID = ?";
// Delete existing data by invoking update action defined in ballerina sql connector
var ret = employeeDB->update(sqlString, employeeID);
match ret {
    int updateRowCount => {
        updateStatus = { "Status": "Data Deleted Successfully" };
    }
    error err => {
        updateStatus = { "Status": "Data Not Deleted", "Error": err.message };
    }
}
return updateStatus;

The endpoint keyword in Ballerina refers to a connection with a remote service. In this case, the remote service is a MySQL database. employeeDB is the reference name for the SQL endpoint. The rest of the code is for preparing SQL queries and executing them by calling the update action in the ballerina/mysql package.

You can implement custom functions in Ballerina that do specific tasks. For this scenario, we have included the following functions to interact with the MySQL database.

Testing
Before you begin
Invoking the employee database service

Add new employee

 -v -X POST -d '{"name":"Alice", "age":20,"ssn":123456789,"employeeId":1}' \
p://localhost:9090/records/employee" -H "Content-Type:application/json"

ut:  
atus":"Data Inserted Successfully"}

Retrieve employee data

 -v  "http://localhost:9090/records/employee/1"

ut: 
mployeeID":1,"Name":"Alice","Age":20,"SSN":123456789}]

Update an existing employee data

 -v -X PUT -d '{"name":"Alice Updated", "age":30,"ssn":123456789,"employeeId":1}' \
p://localhost:9090/records/employee" -H "Content-Type:application/json"

ut: 
atus":"Data Updated Successfully"}

Delete employee data

 -v -X DELETE "http://localhost:9090/records/employee/1"

ut: 
atus":"Data Deleted Successfully"}
Writing unit tests

In Ballerina, the unit test cases should be in the same package inside a folder named as 'test'. When writing the test functions the below convention should be followed.

Deployment

Once you are done with the development, you can deploy the service using any of the methods that are listed below.

Deploying locally

You can deploy the RESTful service that you developed above in your local environment. You need to have the Ballerina installed on your local machine. To deploy simply execute all the following steps.

ballerina build data_backed_service

ballerina run target/data_backed_service.balx 

allerina: deploying service(s) in 'data_backed_service'
allerina: started HTTP/WS server connector 0.0.0.0:9090
Deploying on Docker

You can run the service that we developed above as a docker container. As Ballerina platform includes Ballerina_Docker_Extension, which offers native support for running ballerina programs on containers, you just need to put the corresponding docker annotations on your service code. Since this guide requires MySQL as a prerequisite, you need a couple of more steps to configure MySQL in docker container.

First let's see how to configure MySQL in docker container.

Now let's add the required docker annotations in our employee_db_service. You need to import ballerinax/docker; and add the docker annotations as shown below to enable docker image generation during the build time.

employee_db_service.bal 
ther imports
rt ballerinax/docker;

mployee type definition

reate SQL endpoint to MySQL database
oint mysql:Client employeeDB {
host:<MySQL_Container_IP>,
port:3306,
name:"EMPLOYEE_RECORDS",
username:"root",
password:"root",
poolOptions:{maximumPoolSize:5}


ker:Config {
registry:"ballerina.guides.io",
name:"employee_database_service",
tag:"v1.0"


ker:CopyFiles {
files:[{source:<path_to_JDBC_jar>,
        target:"/ballerina/runtime/bre/lib"}]


ker:Expose {}

oint http:Listener listener {
port:9090


p:ServiceConfig {
basePath:"/records"

ice<http:Service> EmployeeData bind listener {

docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' <Container_ID>

ballerina build data_backed_service

un following command to start docker container: 
ocker run -d -p 9090:9090 ballerina.guides.io/employee_database_service:v1.0

docker run -d -p 9090:9090 ballerina.guides.io/employee_database_service:v1.0

url -v -X POST -d '{"name":"Alice", "age":20,"ssn":123456789,"employeeId":1}' \
http://localhost:9090/records/employee" -H "Content-Type:application/json"
Deploying on Kubernetes

Since this guide requires MySQL as a prerequisite, you need a couple of more steps to create a MySQL pod and use it with our sample.

First let's look at how we can create a MySQL pod in kubernetes.

Now we need to import ballerinax/kubernetes; and use @kubernetes annotations as shown below to enable kubernetes deployment for the service we developed above.

employee_db_service.bal 
age data_backed_service;

ther imports
rt ballerinax/kubernetes;

mployee type definition

reate SQL endpoint to MySQL database
oint mysql:Client employeeDB {
host:"mysql-service",
port:3306,
name:"EMPLOYEE_RECORDS",
username:"root",
password:"root",
poolOptions:{maximumPoolSize:5}


ernetes:Ingress {
hostname:"ballerina.guides.io",
name:"ballerina-guides-employee-database-service",
path:"/"


ernetes:Service {
serviceType:"NodePort",
name:"ballerina-guides-employee-database-service"


ernetes:Deployment {
image:"ballerina.guides.io/employee_database_service:v1.0",
name:"ballerina-guides-employee-database-service",
copyFiles:[{target:"/ballerina/runtime/bre/lib",
            source:<path_to_JDBC_jar>}]


oint http:Listener listener {
port:9090


p:ServiceConfig {
basePath:"/records"

ice<http:Service> EmployeeData bind listener {      

ballerina build data_backed_service

un following command to deploy kubernetes artifacts:  
ubectl apply -f ./target/data_backed_service/kubernetes

kubectl apply -f ./target/data_backed_service/kubernetes 

eployment.extensions "ballerina-guides-employee-database-service" created
ngress.extensions "ballerina-guides-employee-database-service" created
ervice "ballerina-guides-employee-database-service" created

kubectl get service
kubectl get deploy
kubectl get pods
kubectl get ingress

Node Port:

url -v -X POST -d '{"name":"Alice", "age":20,"ssn":123456789,"employeeId":1}' \
http://localhost:<Node_Port>/records/employee" -H "Content-Type:application/json"

Ingress:

Add /etc/hosts entry to match hostname.

27.0.0.1 ballerina.guides.io

Access the service

url -v -X POST -d '{"name":"Alice", "age":20,"ssn":123456789,"employeeId":1}' \
http://ballerina.guides.io/records/employee" -H "Content-Type:application/json"
Observability

Ballerina is by default observable. Meaning you can easily observe your services, resources, etc. Refer to how-to-observe-ballerina-code for more information. However, observability is disabled by default via configuration. Observability can be enabled by adding following configurations to ballerina.conf file in data-backed-service/guide/.

.observability]

.observability.metrics]
ag to enable Metrics
led=true

.observability.tracing]
ag to enable Tracing
led=true

NOTE: The above configuration is the minimum configuration needed to enable tracing and metrics. With these configurations default values are load as the other configuration parameters of metrics and tracing.

Tracing

You can monitor ballerina services using in built tracing capabilities of Ballerina. We'll use Jaeger as the distributed tracing system. Follow the following steps to use tracing with Ballerina.

Metrics

Metrics and alarts are built-in with ballerina. We will use Prometheus as the monitoring tool. Follow the below steps to set up Prometheus and view metrics for Ballerina database service.

b7a.observability.metrics]
nabled=true
rovider="micrometer"

b7a.observability.metrics.micrometer]
egistry.name="prometheus"

b7a.observability.metrics.prometheus]
ort=9700
ostname="0.0.0.0"
escriptions=false
tep="PT1M"

NOTE: Ballerina will by default have following metrics for HTTP server connector. You can enter following expression in Prometheus UI

Logging

Ballerina has a log package for logging to the console. You can import ballerina/log package and start logging. The following section will describe how to search, analyze, and visualize logs in real time using Elastic Stack.

i) Create a file named logstash.conf with the following content

t {  
ts{ 
 port => 5044 



er {  
k{  
 match => { 
 "message" => "%{TIMESTAMP_ISO8601:date}%{SPACE}%{WORD:logLevel}%{SPACE}
 \[%{GREEDYDATA:package}\]%{SPACE}\-%{SPACE}%{GREEDYDATA:logMessage}"
 }  



ut {  
sticsearch{  
 hosts => "elasticsearch:9200"  
 index => "store"  
 document_type => "store_logs"

ii) Save the above logstash.conf inside a directory named as {SAMPLE_ROOT}\pipeline

iii) Start the logstash container, replace the {SAMPLE_ROOT} with your directory name

cker run -h logstash --name logstash --link elasticsearch:elasticsearch \
--rm -v ~/{SAMPLE_ROOT}/pipeline:/usr/share/logstash/pipeline/ \
044:5044 docker.elastic.co/logstash/logstash:6.2.2

i) Create a file named filebeat.yml with the following content

beat.prospectors:
pe: log
ths:
- /usr/share/filebeat/ballerina.log
ut.logstash:
sts: ["logstash:5044"]

NOTE : Modify the ownership of filebeat.yml file using $chmod go-w filebeat.yml

ii) Save the above filebeat.yml inside a directory named as {SAMPLE_ROOT}\filebeat

iii) Start the logstash container, replace the {SAMPLE_ROOT} with your directory name

cker run -v {SAMPLE_ROOT}/filebeat/filebeat.yml:/usr/share/filebeat/filebeat.yml \
SAMPLE_ROOT}/guide/data_backed_service/ballerina.log:/usr/share\
ebeat/ballerina.log --link logstash:logstash docker.elastic.co/beats/filebeat:6.2.2
This work is supported by the National Institutes of Health's National Center for Advancing Translational Sciences, Grant Number U24TR002306. This work is solely the responsibility of the creators and does not necessarily represent the official views of the National Institutes of Health.