Getting Started

Demo 4.2 (Prev)

Lets start with setting up Kafkawize on your Windows/Linux/Unix systems.

Prerequisites

  • Any IDE like IntelliJ/Netbeans or Eclipse to build the applications.
  • Java installed machines to run the applications
  • Rdbms (like Oracle/MySQL..) or File based

1 Download Kafkawize

Download the latest version (4.5.1). https://kafkawize.com/download/

This bundle includes the libraries of kafkawizeuapi and kafkawizeclusterapi

2 Download Metastore

If Rdbms is used as metastore, download

Mysql : https://dev.mysql.com/downloads/

Oracle : https://www.oracle.com/database/technologies/oracle-database-software-downloads.html

of file based/

Or any other Rdbms is ok.

3 License Key

You will receive it from Kafkawize.

4 KW Cluster Api Application

This Api does the below

  • Receives and responds to calls from UI Api
  • Connects to Kafka Brokers with Kafka AdminClient Api
  • There is no connection to any metastore
  • Swagger enabled

The jar (kafkawizeclusterapi-4.5.1.jar) is available in the downloaded bundle.

5 Configure Kafkawize UI Api Application

This Api does the below

  • Users interact with interface with this Api
  • All the end points in this application either connect to Metastore or Cluster Api or both

Configure application properties:

In application.properties

default port is 9097, if port has to be changed, else

server.port:9097

Set metastore to rdbms (from step 4):

kafkawize.db.storetype=rdbms
  • Install and run Rdbms (like Mysql/Oracle) and create a db schema or database

other params:

# default cluster to synchronize data
kafkawize.syncdata.cluster=DEV

# order of envs
kafkawize.envs.order=DEV,TST,ACC,PRD
kafkawize.request.topics.envs=DEV,TST,ACC,PRD

configure db properties like below:

# Spring JPA properties
spring.datasource.url=jdbc:mysql://localhost:3306/kafkametadb?autoReconnect=true&useUnicode=true&useJDBCCompliantTimezoneShift=true&useLegacyDatetimeCode=false&serverTimezone=UTC
spring.datasource.username=kafkauser
spring.datasource.password=kafkauser123
spring.datasource.driver.class=com.mysql.cj.jdbc.Driver
spring.jpa.properties.hibernate.dialect=org.hibernate.dialect.MySQLDialect

configure db properties for file based like below:

# Spring JPA properties
spring.datasource.url=jdbc:h2:file:C:/Software/KafkaWize/H2Filedb/kafkawizedbos;DB_CLOSE_ON_EXIT=FALSE;DB_CLOSE_DELAY=-1;MODE=MySQL;DATABASE_TO_LOWER=TRUE;
spring.datasource.username=kafkauser
spring.datasource.password=kafkauser123
spring.datasource.driver.class=org.h2.Driver
spring.jpa.properties.hibernate.dialect=org.hibernate.dialect.H2Dialect

Configure Cluster Api

configure cluster api host and port details:

kafkawize.clusterapi.url:http://localhost:9343
  • ignore user/pwd of cluster api properties

6 Metastore setup

Metastore Rdbms

In application.properties configure custom.dbscripts.execution=auto or manual to get the required database scripts executed.

Configure custom.dbscripts.dropall_recreate=false or true to drop all the metadata scripts and recreate. This property is only valid if the above property is set to auto

Configure in application properties:: # Database settings # To create all the required tables of Kafkawize. Need grants to create and alter. Values true/false:

kafkawize.dbscripts.create.tables=false

# db scripts execution ‘auto’ or ‘manual’. If set to manual, user has to execute the scripts from resources dir manually:

kafkawize.dbscripts.insert.basicdata=false

# Location of db scripts for insert or create:

kafkawize.dbscripts.location=./scripts/base/rdbms/

  • Install and run Mysql/Oracle and create a db schema or database

  • Create tables and run insert scripts in Database

    /scripts/base/rdbms/ddl-jdbc.sql

    /scripts/base/rdbms/insertdata.sql

  • Above scripts will create tables and insert initial set of Environments, Teams and Users which you can delete anytime from UI.

The jar (kafkawize-4.5.1.jar) is available in the downloaded bundle.

7 Run KW and KWClusterApi

Run the script to start the application.

Windows : start-kafkawize.bat Linux/Unix : start-kafkawize.sh

Or

Run:

java -jar kafkawizeclusterapi-4.5.1.jar --spring.config.location=classpath:/application.properties

Run:

java -jar kafkawize-4.5.1.jar --spring.config.location=classpath:/application.properties

If application is running, you can access UI from http://[host]:[port]/kafkawize

8 Kafka Connectivity

Cluster Api Application connects to Kafka brokers with Kafka AdminClient Api., and needs Describe access on all topics through the cluster. Hence the below wildcard acl has to be executed.

  • If Acls are enabled on Kafka brokers, make sure “Cluster Api” application host is authorized to read topics (A read Acl is enough on the topic)

    Examples SSL Based Acl (Note of double quotes in the below command if copied properly):

    bin/kafka-acls –authorizer-properties zookeeper.connect=localhost:2181 –add –allow-principal User:CN=MO,OU=MO,O=WA,L=WA,ST=WA,C=HO” –operation All –topic “*” –cluster Cluster:kafka-cluster

    Examples IP Based Acl:

    bin/kafka-acls –authorizer-properties zookeeper.connect=localhost:2181 –add –allow-principal User:”*” –allow-host 127.0.0.1 –operation All –topic “*” –cluster Cluster:kafka-cluster

9 Final Check

  • Cluster Api is running

  • Metastore (Rdbms or file system) is running and has tables and data

  • UI Api is running

  • Cluster Api is authorized to read topics and acls on topics information(Acls should be configured)

  • Access UI from http://[host]:[port]/kafkawize where host and port are UI Api application Example : http://localhost:9097/kafkawize

    Default users, passwords and roles:

    gary/user USER Octopus will/user USER Seahorses john/user USER Starfish cris/user ADMIN Octopus noah/user ADMIN Seahorses alex/user ADMIN Starfish superuser/user SUPERUSER Seahorses