Kafkawize is built with the key features as below:
- 4 eyed principle – Requesting and Approving topics/acls/schemas - Spring based security - Fully automated - Browse Topics, Acls, Schemas, Connectors, Producers and Consumers - Synchronize Source of truth with Meta store, Restore config - Support for an Rdbms or file system as metastore
There are two main applications of Kafkawize. UiApi and ClusterApi. There is a security applied on Cluster Api. End points are exposed to anyone who can reach the host and has access., and swagger is enabled too. Users can directly post/get requests to this application.
On the UI Api, Spring security is enabled. As the application boots, it loads all the users from database into its memory and would be referred to it.
Security on the Kafka Broker is out of scope for Kafkawize., , however for Kafkawize to connect to Kafka broker, Acls/AdminClient properties need to be provided to avoid authorization issues.
Teams and Role Management¶
Kafkawize can be configured with any number of roles and permissions:
By default ‘USER’ roles can request and approve topic/acl/schema/connector requests.
Portal can be accessed only if the user logs in. New users are created by Super users.
After a user is logged in, a session is created for the user, and depending on the role , he/she is authorized to perform certain actions.
Default login url : http://localhost:9097/kafkawize
After a user is logged in , he/she will be redirected to homepage.
All the logged in users with any roles can browse the below components.
A user is allowed to a select an environment, and the topics are loaded. If you do not see the topics for some reason, check for any errors in Cluster Api application and UI Api application.
A filter option is also provided and users can search for specific topics.
Every topic has a topic overview page, which displays topic partitions, replication factor in all available environments.
Every topic has a topic overview page, which displays topic subscriptions in all available environments.
Note : Make sure kafkawize has access to read topics (Acl is required if acls are enabled on the cluster)
Acl - Access Controls
From the Browse topics screen, user can select and clicking on a topic displays all the available subscriptions of that topic.
Acls are either Producers or Consumers. Producers have write access on the topics and Consumers have read access on the topics.
Acls are either IP Address based or SSL based. If IP based, you should see the IP addresses of client who has access. If SSL based, the DN name should be seen.
Every Acl should be owned by a team. If no team is assigned to an Acl, it can be assigned by Synchronize Acl option, but by a Super user.
There can be multiple producers and multiple consumers for a topic owning by different teams.
All the users can request for topics, acls or avro schemas in any environment.
Topic Requests can be approved by Admins or Super users from the same team.
Acl(Subscription) Requests can be approved by Admins or Super users who belong to Topic Owner’s team.
Schema Requests can be approved by Admins or Super users from the same team.
- Notifications: After every request is raised, there is a notification (bell icon) at the top for the relevant user who can approve these requests.
To request for a topic, all the mandatory parameters need to be provided.
Topic name, partitions, owning team, application name, and if any remarks. On submission you can view your requests in Audit/My Requests., and it can be deleted if required.
Default partitions size, maximum partitions size and default replication factor can be configured per environment in Clusters page.
To request for a topic in higher environments, it is required for the topic to exist in Base Sync cluster (DEV). This can be configured with property custom.syncdata.cluster
To request for a acl, all the mandatory parameters need to be provided.
After a user clicks on a topic, user can view all the subscriptions and a button to subscribe to the topic.
The below fields are required.
Acl type Producer or Consumer.
IP Addresses or SSL DN name, or username should be provided for authorizations.
Multiple IP addresses, or principles or usernames can be filled at once
On submission you can view your requests in Audit/My Requests., and it can be deleted if required.
An Avro schema can also be requested for a particular subject (topic).
All the clusters are of type ‘kafka’ or ‘schemaregistry’.
New clusters can be added by Superusers only. There is a other parameters field added, into which default partitions, max partitions size and replication factor can be added.
A cluster can be deleted by any Admin user or super user.
All the requests from users are audited and can be viewed.
My Topic Requests¶
Requests (Topics/Acls/Schemas/Connectors) from their own team can be viewed and deleted.
My Acl Requests¶
Requests (Topics/Acls/Schemas) from their own team can be viewed and deleted.
|width:||500px :align: center|
All the requests requested and approved can be viewed. Users login/logout are not part of this log yet.
Approve Or Decline¶
Topic Requests can be approved or declined by users from the same team. After approval of a topic, it is created on the cluster, how ever no acls are assigned to it yet.
Acl Requests can be approved/declined by users who belong to Topic Owner’s team. After approval, acls are created on the cluster.
Schema Requests can be approved by Admins or Super users from the same team.
Notifications are visible on the top right side of the portal.
If a topic is requested by ‘user1’ from ‘Team1’, it can be approved/declined by ‘user2’ from ‘Team1’ only., provided ‘user2’ has same role.
If a acl is requested by ‘user1’ on topic which is owned by ‘Team2’, it can be approved/declined by ‘user2’ from ‘Team2’ only., provided ‘user2’ has same role.
If a schema is requested by ‘user1’ from ‘Team1’, it can be approved by ‘user2’ from ‘Team1’ only., provided ‘user2’ has same role.
All the users are visible to any logged in user. A new user can be added/deleted by only Super users.
From this page all the users can be seen and also be deleted.
With this form a new user can be added by a SuperUser.
All the teams are visible to any logged in user. A new team can be added, or a team can be deleted by only Super users.
From this page all the users can be seen.
With this form a new team can be added by a SuperUser.
All the Server configuration including custom properties (application properties can be viewed)
All Users can view their profile with the button available in the top right corner of the portal.
All the logged in users can change the password, however it will be effective only after the UI Api Application is restarted. Enhancement will be provided on this soon.
Users can logout after clicking on the logout button on the top right corner. Session will be killed after this action.
If a topic exists in Dev environment, it can be requested for promotion to the next higher environment (TST). If topic exists in Dev and Tst environments, promotion can be requested for Acceptance environment, with the same topic name. This way it restricts users to create topics in adhoc way and at the same time maintains data integrity. This feature provides an easy way to request for topics in higher environments, keeping partitions, replication factor parameters specific to those environments. This deletion request can be triggered only by Topic owner teams. Please note, it follows the same request/approval process.
When a topic is no more needed,it is best to get it deleted. So we save on the partitions loaded by the cluster. This feature makes sure a topic can only be deleted when there are no subscriptions for the topic. Hence, no Producers and Consumers are affected by deleting. This deletion request can be triggered only by Topic owner teams. Please note, it follows the same request/approval process.
It is possible to delete particular Producer or Consumer subscriptions of a topic with this feature. This deletion request can be triggered by that Acl owner teams. No other teams can request for other’s acl deletion. So all the subscriptions are secure this way.
Ldap Authentication/Active Directory¶
Users can be authenticated against Ldap/Active Directory while users login to Kafkawize.
If you do not want to create/use a different password for Kafkawize users, rather authenticate from an existing Ldap server of your organization, it is easy to integrate now. Application properties have properties like baseDN, userDNPattern, passwordAttributes etc to configure your Ldap server and connect. The below property enables ldap authentication:
Cluster Connect Protocols¶
kafkawizeclusterapi can connect to Kafka cluster over the below protocols.
Kafkawizeclusterapi connects can connect to Kafka clusters over PLAINTEXT protocol When a cluster is being created in Kafkawize, this protocol can be selected.
Kafkawizeclusterapi connects can connect to Kafka clusters over SSL protocol When a cluster is being created in Kafkawize, this protocol can be selected. Make sure SSL parameters are configured in application.properties in Kafkawizeclusterapi. SSL Parameters like keystore, truststore, passwords, etc.
Kafkawizeclusterapi connects can connect to Kafka clusters over SASL_PlAIN protocol When a cluster is being created in Kafkawize, this protocol can be selected. Make sure SSL parameters are configured in application.properties in Kafkawizeclusterapi.
SASL stands for Simple Authentication and Security Layer. Kafka Jaas configuration has to be configured on the Kafka cluster with username/passwords. Kafkawizeclusterapi application.properties would be configured with SASL Jaas config too like below:
org.apache.kafka.common.security.plain.PlainLoginModule required username='admin' password='admin-secret';
Kafkawizeclusterapi connects can connect to Kafka clusters over SASL_SSL protocol When a cluster is being created in Kafkawize, this protocol can be selected. Make sure SSL parameters are configured in application.properties in Kafkawizeclusterapi.
SASL stands for Simple Authentication and Security Layer. Kafka Jaas configuration has to be configured on the Kafka cluster with username/passwords. Kafkawizeclusterapi application.properties would be configured with SASL Jaas config too like below
It is possible to configure PlainLoginModule or KerberosLoginModule:
com.sun.security.auth.module.Krb5LoginModule required \ useKeyTab=true \ storeKey=true \ keyTab="/etc/security/keytabs/kafka_client.keytab" \ principal="kafkaclient1@EXAMPLE.COM";
SSL parameters should also be configured, when ssl encryption is enabled.
SASL mechanism can be PLAIN or GSSAPI(Kerberos)
A situation where Kafka cluster already exists and would like to adapt Kafkawize in your organization, all the topics and acls need to have their teams/owners.
This feature is possible with Synchronization of Topics or Acls.
After a environment is selected, topics are displayed, and a team can be assigned to it. And this action, team becomes the Owner team.
It is required to synchronize the topic team first with Base sync cluster first. Base sync cluster can be configured with property custom.syncdata.cluster in application properties.
After a environment is selected, Producer and Consumer Acls are displayed, and a team can be assigned to it. After this action, team becomes the Owner of that subscription (producer or consumer).
Configuration of topics and acls from metastore can be restored back on any selected Kafka cluster.
Each tenant can manage their topics with their own teams in isolation. Every tenant can have their own set of Kafka environments, and users of one tenant cannot view/access topics, acls or any from other tenants. It provides an isolation avoiding any security breach.
Several charts are introduced to give an overview of Clusters and usages by teams. Topics, Acls and Partitions per team Topics, Acls and Partitions per environments Activity log per team and per environments
Configurable Roles & Permissions¶
Any new roles can be added and associate different kind of permissions Permissions A whole bunch of permissions can be assigned to roles from User interface, making it very flexible. With immediate effect, users will be able to see the changes. Ex: A permission to request topics, or approve or add users, environments, clusters etc.
New users can register from the home page, and request is forwarded to Super user. After the approval, user is added to the system.
For every request and approval, through out the Kafkawize application, emails are sent out to approvers, and after approvals, notify the requesters. When a new user is added, or password changes, etc, notifications are enabled.
Create and approve Kafka connectors