Confluent REST Proxy

Confluent REST Proxy is a component of the Confluent Platform that provides a RESTful interface to interact with Apache Kafka. It allows you to produce and consume messages, create and manage topics, and perform other Kafka operations using HTTP requests. REST Proxy simplifies the integration of Kafka with applications and systems that may not have native Kafka clients available.

Key Features

1. RESTful API: REST Proxy exposes a RESTful API for interacting with Kafka. It supports HTTP methods like POST, GET, and DELETE to perform various operations on Kafka resources.

2. Message Production: You can produce messages to Kafka topics by sending POST requests to the REST Proxy endpoint. The messages can be sent in JSON or binary format.

3. Message Consumption: REST Proxy allows you to consume messages from Kafka topics using GET requests. You can retrieve messages based on offsets or timestamps.

4. Topic Management: REST Proxy provides endpoints to create, delete, and retrieve information about Kafka topics. You can easily manage your Kafka topics through the RESTful API.

5. Schema Registry Integration: REST Proxy integrates with Confluent Schema Registry, allowing you to work with Avro-serialized data. It can automatically register and retrieve schemas for the messages being produced or consumed.

Getting Started

To get started with Confluent REST Proxy, follow these steps:

1. Installation: Install the Confluent Platform, which includes the REST Proxy component. You can download the platform from the Confluent website and follow the installation instructions for your operating system.

2. Configuration: Configure the REST Proxy by modifying the kafka-rest.properties file. Specify the Kafka broker endpoints, listener port, and other required settings.

3. Start the REST Proxy: Start the REST Proxy service using the provided start script. For example:

   $ <path-to-confluent>/bin/kafka-rest-start <path-to-confluent>/etc/kafka-rest/kafka-rest.properties
   

4. API Requests: Once the REST Proxy is running, you can interact with Kafka using HTTP requests. Here are a few examples:

   • Produce a message to a topic:

     POST /topics/my-topic
     Content-Type: application/vnd.kafka.json.v2+json
     
     {
       "records": [
         {
           "key": "key1",
           "value": "value1"
         },
         {
           "key": "key2",
           "value": "value2"
         }
       ]
     }
     

   • Consume messages from a topic:

     GET /consumers/my-group/instances/my-instance/records?timeout=3000&max_bytes=300000
     

   • Create a new topic:

     POST /topics
     Content-Type: application/json
     
     {
       "topic_name": "new-topic",
       "partitions_count": 3,
       "replication_factor": 1
     }
     

For detailed API documentation and more examples, refer to the Confluent REST Proxy documentation.

Benefits

Confluent REST Proxy offers several benefits:

1. Language Agnostic: REST Proxy allows you to interact with Kafka using standard HTTP requests, making it compatible with any programming language or system that supports HTTP.

2. Simplified Integration: REST Proxy simplifies the integration of Kafka with external systems, such as web applications, mobile apps, or data pipelines, by providing a familiar RESTful interface.

3. Reduced Dependency: With REST Proxy, you don’t need to include Kafka client libraries in your application code. This reduces the dependency on Kafka-specific libraries and simplifies the application development process.

4. Improved Security: REST Proxy supports authentication and authorization mechanisms, such as SSL/TLS and SASL, to secure the communication between clients and Kafka.