Getting Started

Lets start with setting up Kafkawize on your Windows/Linux/Unix systems.

Prerequisites

  • Any IDE like IntelliJ/Netbeans or Eclipse to build the applications.
  • Java installed machines to run the applications
  • Rdbms (like Oracle/MySQL..) or Apache Cassandra

1 Download Kafkawize 3.1

Download the latest version (3.1) or clone from the git repo. https://github.com/muralibasani/kafkawize

Download the latest version (3.1) or clone from the git repo. https://github.com/muralibasani/kafkawizeclusterapi

OR

Download Kafkawize 3.1 bundle here Kafkawize_3.1

2 Download Metastore

If Cassandra is used as metastore, you can download from http://cassandra.apache.org/download/

If Rdbms is used as metastore, download

Mysql : https://dev.mysql.com/downloads/

Oracle : https://www.oracle.com/database/technologies/oracle-database-software-downloads.html

Or any other Rdbms is ok.

3 Build KW Cluster Api Application

This Api does the below

  • Receives and responds to calls from UI Api
  • Connects to Kafka Brokers with Kafka AdminClient Api
  • There is no connection to any metastore
  • Swagger enabled

Unzip kafkawizeclusterapi.zip and open in any IDE

Configure the application properties (src/main/resources) if port has to be changed, else default port is 9343

and run maven command to create a runnable jar:

mvn clean package

This should create a jar (kafkawizeclusterapi-3.1.jar) in target dir.

Run:

java -jar kafkawizeclusterapi-3.1.jar

4 Metastore setup

Metastore Cassandra

  • Install and run Cassandra and create a keyspace ‘kafkamanagementapi’

  • Create tables and run insert scripts in Cassandra

    /kafkawize/kafkawize-web/src/main/resources/scripts/base/cassandra/createcassandra.sql

    /kafkawize/kafkawize-web/src/main/resources/scripts/base/cassandra/insertdata.sql

    (Scripts available in kafkawize.zip)

  • Above scripts will create tables and insert initial set of Environments, Teams and Users which you can delete anytime from UI.

Metastore Rdbms

  • Install and run Mysql/Oracle and create a db schema or database

  • Create tables and run insert scripts in Database

    /kafkawize/kafkawize-web/src/main/resources/scripts/base/rdbms/ddl-jdbc.sql

    /kafkawize/kafkawize-web/src/main/resources/scripts/base/rdbms/insertdata.sql

    (Scripts available in kafkawize.zip)

  • Above scripts will create tables and insert initial set of Environments, Teams and Users which you can delete anytime from UI.

5 Build KW UI Api Application

This Api does the below

  • Users interact with interface with this Api
  • All the end points in this application either connect to Metastore or Cluster Api or both

Unzip kafkawize.zip and open in any IDE

Configure application properties:

In kafkawize/kafkawize-conf/environments/{local/test/prod}/application.properties

default port is 9097, if port has to be changed, else

server.port:9097

If metastore is cassandra (from step 4)

setstore type as cassandra:

db.storetype=cassandra

configure cassandra host, port and keyspace:

cassandradb.keyspace:kafkamanagementapi
cassandradb.url:localhost
cassandradb.port:9042

If metastore is rdbms (from step 4)

setstore type as rdbms:

db.storetype=rdbms
  • Install and run Rdbms (like Mysql/Oracle) and create a db schema or database

configure db properties like below:

# Spring JPA properties
spring.datasource.url=jdbc:mysql://localhost:3306/kafkametadb?autoReconnect=true&useUnicode=true&useJDBCCompliantTimezoneShift=true&useLegacyDatetimeCode=false&serverTimezone=UTC
spring.datasource.username=kafkauser
spring.datasource.password=kafkauser123
spring.datasource.driver.class=com.mysql.cj.jdbc.Driver

Configure Cluster Api

configure cluster api host and port details:

clusterapi.url:http://localhost:9343
  • ignore user/pwd of cluster api properties

Build

Run maven command to create a runnable jar:

mvn clean package

This should create a jar in target dir (kafkawizekafkawize-webtargetkafkawize-web-3.1.jar).

Run:

java -jar spring.config.location=classpath:/application.properties -Dspring.profiles.active=[local/test/prod] kafkawizeclusterapi-3.1.jar

If application is running, you can access UI from http://[host]:[port]/kafkawize

6 Kafka Connectivity

Cluster Api Application connects to Kafka brokers with Kafka AdminClient Api.

  • If Acls are enabled on Kafka brokers, make sure Cluster Api application host is authorized to read topics (A read Acl is enough on the topic)
  • If SASL/SSL is configured, make sure they right properties are configured in AdminClient properties in Cluster Api application.

7 Final Check