Developer steering on the way to do native testing with Amazon MSK Serverless -

Amazon Managed Streaming for Apache Kafka (Amazon MSK) is a totally managed service that makes it simple to construct and run Kafka clusters on Amazon Internet Providers (AWS). When working with Amazon MSK, builders are focused on accessing the service regionally. This enables builders to check their utility with a Kafka cluster that has the identical configuration as manufacturing and offers an an identical infrastructure to the precise surroundings without having to run Kafka regionally.

An Amazon MSK Serverless personal DNS endpoint is barely accessible from Amazon Digital Personal Cloud (Amazon VPC) connections which have been configured to attach. It isn’t immediately resolvable out of your native growth surroundings. One possibility is to make use of AWS Direct Join or AWS VPN to have the ability to Hook up with Amazon MSK Serverless out of your on-premises community. Nevertheless, constructing such an answer might incur price and complexity, and it must be arrange by a platform staff.

This submit presents a sensible strategy to accessing your Amazon MSK surroundings for growth functions by means of a bastion host utilizing a Safe Shell (SSH) tunnel (a generally used safe connection technique). Whether or not you’re working with Amazon MSK Serverless, the place public entry is unavailable, or with provisioned MSK clusters which can be deliberately saved personal, this submit guides you thru the steps to ascertain a safe connection and seamlessly combine your native growth surroundings together with your MSK assets.

Answer overview

The answer permits you to immediately hook up with the Amazon MSK Serverless service out of your native growth surroundings with out utilizing Direct Join or a VPN. The service is accessed with the bootstrap server DNS endpoint boot-<<xxxxxx>>.c<<x>>.kafka-serverless.<<region-name>>.amazonaws.com on port 9098, then routed by means of an SSH tunnel to a bastion host, which connects to the MSK Serverless cluster. Within the subsequent step, let’s discover the way to arrange this connection.

The move of the answer is as follows:

The Kafka consumer sends a request to connect with the bootstrap server
The DNS question to your MSK Serverless endpoint is routed to a regionally configured DNS server
The regionally configured DNS server routes the DNS question to localhost.
The SSH tunnel forwards all of the visitors on port 9098 from the localhost to the MSK Serverless server by means of the Amazon Elastic Compute Cloud (Amazon EC2) bastion host.

The next picture exhibits the structure diagram.

Stipulations

Earlier than deploying the answer, you must have the next assets deployed in your account:

An MSK Serverless cluster configured with AWS Identification and Entry Administration (IAM) authentication.
A bastion host occasion with community entry to the MSK Serverless cluster and SSH public key authentication.
AWS CLI configured with an IAM person and capable of learn and create matters on Amazon MSK. Use the IAM coverage from Step 2: Create an IAM position within the Getting began utilizing MSK Serverless clusters
For Home windows customers, set up Linux on Home windows with Home windows Subsystem for Linux 2 (WSL 2) utilizing Ubuntu 24.04. For steering, check with Find out how to set up Linux on Home windows with WSL.

This information assumes an MSK Serverless deployment in us-east-1, however it may be utilized in each AWS Area the place MSK Serverless is offered. Moreover, we’re utilizing OS X as working system. Within the following steps change msk-endpoint-url together with your MSK Serverless endpoint URL with IAM authentication. The MSK endpoint URL has a format like boot-<<xxxxxx>>.c<<x>>.kafka-serverless.<<region-name>>.amazonaws.com.

Answer walkthrough

To entry your Amazon MSK surroundings for growth functions, use the next walkthrough.

Configure native DNS server OSX

Set up Dnsmasq as a neighborhood DNS server and configure the resolver to resolve the Amazon MSK. The answer makes use of Dnsmasq as a result of it may well examine DNS requests in opposition to a database of patterns and use these to find out the proper response. This performance can match any request that ends in kafka-serverless.us-east-1.amazonaws.com and ship 127.0.0.1 in response. Comply with these steps to put in Dnsmasq:

Replace brew and set up Dnsmasq utilizing brew
```
brew up
brew set up dnsmasq
```
Begin the Dnsmasq service
```
sudo brew providers begin dnsmasq
```

Reroute all visitors for Serverless MSK (kafka-serverless.us-east-1.amazonaws.com) to 127.0.0.1

echo deal with=/kafka-serverless.us-east-1.amazonaws.com/127.0.0.1 >> $(brew --prefix)/and so forth/dnsmasq.conf

Reload Dnsmasq configuration and clear cache

sudo launchctl unload /Library/LaunchDaemons/homebrew.mxcl.dnsmasq.plist
sudo launchctl load /Library/LaunchDaemons/homebrew.mxcl.dnsmasq.plist
dscacheutil -flushcache

Configure OS X resolver

Now that you’ve a working DNS server, you may configure your working system to make use of it. Configure the server to ship solely .kafka-serverless.us-east-1.amazonaws.com queries to Dnsmasq. Most working methods which can be just like UNIX have a configuration file referred to as /and so forth/resolv.conf that controls the way in which DNS queries are carried out, together with the default server to make use of for DNS queries. Use the next steps to configure the OS X resolver:

OS X additionally permits you to configure further resolvers by creating configuration recordsdata within the /and so forth/resolver/ This listing in all probability gained’t exist in your system, so your first step ought to be to create it:
```
sudo mkdir -p /and so forth/resolver
```
Create a brand new file with the identical identify as your new top-level area (kafka-serverless.us-east-1.amazonaws.com) within the /and so forth/resolver/ listing and add 127.0.0.1 as a nameserver to it by getting into the next command.
```
sudo tee /and so forth/resolver/kafka-serverless.us-east-1.amazonaws.com >/dev/null <<EOF
nameserver 127.0.0.1
EOF
```

Configure native DNS server Home windows

In Home windows Subsystem for Linux, first set up Dnsmasq, then configure the resolver to resolve the Amazon MSK and eventually add localhost as the primary nameserver.

Replace apt and set up Dnsmasq utilizing apt. Set up the telnet utility for later assessments:
```
sudo apt replace
sudo apt set up dnsmasq
sudo apt set up telnet
```

Reroute all visitors for Serverless MSK (kafka-serverless.us-east-1.amazonaws.com) to 127.0.0.1.

echo "deal with=/kafka-serverless.us-east-1.amazonaws.com/127.0.0.1" | sudo tee -a /and so forth/dnsmasq.conf

Reload Dnsmasq configuration and clear cache.
```
sudo /and so forth/init.d/dnsmasq restart
```
Open /and so forth/resolv.conf and add the next code within the first line.
```
nameserver 127.0.0.1
```
The output ought to seem like the next code.
```
#Some feedback
nameserver 127.0.0.1
nameserver <<your_nameservers>>
..
```

Create SSH tunnel

The subsequent step is to create the SSH tunnel, which is able to permit any connections made to localhost:9098 in your native machine to be forwarded over the SSH tunnel to the goal Kafka dealer. Use the next steps to create the SSH tunnel:

Exchange bastion-host-dns-endpoint with the general public DNS endpoint of the bastion host, which comes within the model of <<xyz>>.compute-1.amazonaws.com, and change ec2-key-pair.pem with the important thing pair of the bastion host. Then create the SSH tunnel by getting into the next command.
```
ssh -i "~/<<ec2-key-pair.pem>>" ec2-user@<<bastion-host-dns-endpoint>> -L 127.0.0.1:9098:<<msk-endpoint-url>>:9098
```
Go away the SSH tunnel operating and open a brand new terminal window.

Take a look at the connection to the Amazon MSK server by getting into the next command.

telnet <<msk-endpoint-url>> 9098

The output ought to seem like the next instance.

Attempting 127.0.0.1...
Linked to boot-<<xxxxxxxx>>.c<<x>>.kafka-serverless.us-east-1.amazonaws.com.
Escape character is '^]'.

Testing

Now configure the Kafka consumer to make use of IAM Authentication after which check the setup. You discover the newest Kafka set up on the Apache Kafka Obtain web site. Then unzip and duplicate the content material of the Dafka folder into ~/kafka.

Obtain the IAM authentication and unpack it

cd ~/kafka/libs
wget https://github.com/aws/aws-msk-iam-auth/releases/obtain/v2.2.0/aws-msk-iam-auth-2.2.0-all.jar
cd ~

Configure Kafka properties to make use of IAM because the authentication mechanism

cat <<EOF > ~/kafka/config/client-config.properties

# Units up TLS for encryption and SASL for authN.

safety.protocol = SASL_SSL

# Identifies the SASL mechanism to make use of.

sasl.mechanism = AWS_MSK_IAM

# Binds SASL consumer implementation.

sasl.jaas.config = software program.amazon.msk.auth.iam.IAMLoginModule required;


# Encapsulates developing a SigV4 signature primarily based on extracted credentials.

# The SASL consumer sure by "sasl.jaas.config" invokes this class.

sasl.consumer.callback.handler.class = software program.amazon.msk.auth.iam.IAMClientCallbackHandler

EOF

Enter the next command in ~/kafka/bin to create an instance subject. Ensure that the SSH tunnel created within the earlier part remains to be open and operating.

./kafka-topics.sh --bootstrap-server <<msk-endpoint-url>>:9098 --command-config ~/kafka/config/client-config.properties --create --topic ExampleTopic --partitions 10 --replication-factor 3 --config retention.ms=3600000

Cleanup

To take away the answer, full the next steps for Mac customers:

Delete the file /and so forth/resolver/kafka-serverless.us-east-1.amazonaws.com
Delete the entry deal with=/kafka-serverless.us-east-1.amazonaws.com/127.0.0.1 within the file $(brew --prefix)/and so forth/dnsmasq.conf
Cease the Dnsmasq service sudo brew providers cease dnsmasq
Take away the Dnsmasq service sudo brew uninstall dnsmasq

To take away the answer, full the next steps for WSL customers:

Delete the file /and so forth/dnsmasq.conf
Delete the entry nameserver 127.0.0.1 within the file /and so forth/resolv.conf
Take away the Dnsmasq service sudo apt take away dnsmasq
Take away the telnet utility sudo apt take away telnet

Conclusion

On this submit, I offered you with steering on how builders can hook up with Amazon MSK Serverless from native environments. The connection is completed utilizing an Amazon MSK endpoint by means of an SSH tunnel and a bastion host. This permits builders to experiment and check regionally, without having to setup a separate Kafka cluster.

Concerning the Creator

Simon Peyer is a Options Architect at Amazon Internet Providers (AWS) primarily based in Switzerland. He’s a sensible doer and enthusiastic about connecting know-how and other people utilizing AWS Cloud providers. A particular focus for him is knowledge streaming and automations. Moreover work, Simon enjoys his household, the outside, and climbing within the mountains.

Developer steering on the way to do native testing with Amazon MSK Serverless