In Cloudera deployments on public cloud, one of many key configuration parts is the DNS. Get it mistaken and your deployment might turn into wholly unusable with customers unable to entry and use the Cloudera knowledge companies. If the DNS is about up much less ultimate than it might be, connectivity and efficiency points might come up. On this weblog, we’ll take you thru our tried and examined greatest practices for organising your DNS to be used with Cloudera on Azure.
To get began and offer you a really feel for the dependencies for the DNS, in an Azure deployment for Cloudera, these are the Azure managed companies getting used:
- AKS cluster: knowledge warehouse, knowledge engineering, machine studying, and Knowledge circulation
- MySQL database: knowledge engineering
- Storage account: all companies
- Azure database for PostgreSQL DB: knowledge lake and knowledge hub clusters
- Key vault: all companies
Typical buyer governance restrictions and the influence
Most Azure customers use personal networks with a firewall as egress management. Most customers have restrictions on firewalls for wildcard guidelines. Cloudera assets are created on the fly, which suggests wildcard guidelines could also be declined by the safety staff.
Most Azure customers use hub-spoke community topology. DNS servers are often deployed within the hub digital community or an on-prem knowledge heart as an alternative of within the Cloudera VNET. Which means if DNS isn’t configured appropriately, the deployment will fail.
Most Cloudera prospects deploying on Azure enable the usage of service endpoints; there’s a smaller set of organizations that don’t enable the usage of service endpoints. Service endpoint is an easier implementation to permit assets on a personal community to entry managed companies on Azure Cloud. If service endpoints usually are not allowed, firewall and personal endpoints would be the different two choices. Most cloud customers don’t like opening firewall guidelines as a result of that can introduce the chance of exposing personal knowledge on the web. That leaves personal endpoints the one choice, which may also introduce further DNS configuration for the personal endpoints.
Connectivity from personal community to Azure managed companies
Firewall to Web
Route from firewall to Azure managed service endpoint on the web instantly.
Service endpoint
Azure gives service endpoints for assets on personal networks to entry the managed companies on the web with out going by means of the firewall. That may be configured at a subnet degree. Since Cloudera assets are deployed in numerous subnets, this configuration should be enabled on all subnets.
The DNS data of the managed companies utilizing service endpoints can be on the web and managed by Microsoft. The IP tackle of this service can be a public IP, and routable from the subnet. Please discuss with the Microsoft documentation for element.
Not all managed companies help companies endpoint. In a Cloudera deployment state of affairs, solely storage accounts, PostgreSQL DB, and Key Vault help service endpoints.
Luckily, most customers enable service endpoints. If a buyer doesn’t enable service endpoints, they should go together with a personal endpoint, which has similarities to what must be configured within the following content material.
Non-public Endpoint
There’s a community interface with a personal IP tackle created with a personal endpoint, and there’s a personal hyperlink service related to a selected community interface, in order that different assets within the personal community can entry this service by means of the personal community IP tackle.
The important thing right here is for the personal assets to discover a DNS resolve for that non-public IP tackle. There are two choices to retailer the DNS document:
- Azure managed public DNS zones will all the time be there, however they retailer several types of IP addresses for the personal endpoint. For instance:
- Storage account personal endpoint—the general public DNS zone shops the general public IP tackle of that service.
- AKS API server personal endpoint—the general public DNS zone shops the personal IP of that service.
- Azure Non-public DNS zone: The DNS data can be synchronized to the Azure Default DNS of LINKED VNET.
Non-public endpoint is eligible to all Azure managed companies which might be utilized in Cloudera deployments.
As a consequence, for storage accounts, customers both use service endpoints or personal endpoints. As a result of the general public DNS zone will all the time return a public IP, the personal DNS zone turns into a compulsory configuration.
For AKS, these two DNS alternate options are each appropriate. The challenges of personal DNS zones can be mentioned subsequent.
Challenges of personal DNS zone on Azure personal community
Essential Assumptions
As talked about above for the standard state of affairs, most Azure customers are utilizing a hub-and-spoke community structure, and deploy customized personal DNS on hub VNET.
The DNS data can be synchronized to Azure default DNS of linked VNET.
Easy Structure Use Instances
One VNET state of affairs with personal DNS zone:
When a personal endpoint is created, Cloudera on Azure will register the personal endpoint to the personal DNS zone. The DNS document can be synchronized to Azure Default DNS of linked VNET.
If customers use customized personal DNS, they will configure conditional ahead to Azure Default DNS for the area suffix of the FQDN.
Hub-and-spoke VNET with Azure default DNS:
With hub-spoke VNET and Azure default DNS, that’s nonetheless acceptable. The one drawback is that the assets on the un-linked VNET will be unable to entry the AKS. However since AKS is utilized by Cloudera, that doesn’t pose any main points.
The Problem Half
The preferred community structure amongst Azure customers is hub-spoke community with customized personal DNS servers deployed both on hub-VNET or on-premises community.
Since DNS data usually are not synchronized to the Azure Default DNS of the hub VNET, the customized personal DNS server can’t discover the DNS document for the personal endpoint. And since the Cloudera VNET is utilizing the customized personal DNS server on hub VNET, the Cloudera assets on Cloudera VNET will go to a customized personal DNS server for DNS decision of the FQDN of the personal endpoint. The provisioning will fail.
With the DNS server deployed within the on-prem community, there isn’t Azure default DNS related to the on-prem community, so the DNS server couldn’t discover the DNS document of the FQDN of the personal endpoint.
Configuration greatest practices
Towards the background
Possibility 1: Disable Non-public DNS Zone
Use Azure managed public DNS zone as an alternative of a personal DNS zone.
- For knowledge warehouse: create knowledge warehouses by means of the Cloudera command line interface with the parameter “privateDNSZoneAKS”: set to”None.”
- For Liftie-based knowledge companies: the entitlement “LIFTIE_AKS_DISABLE_PRIVATE_DNS_ZONE” should be set. Clients can request this entitlement to be set both by means of a JIRA ticket or have their Cloudera answer engineer to make the request on their behalf.
The only disadvantage of this feature is that it doesn’t apply to knowledge engineering, since that knowledge service will create and use a MySQL personal DNS zone on the fly. There may be at current no choice to disable personal DNS zones for knowledge engineering.
Possibility 2: Pre-create Non-public DNS Zones
Pre-create personal DNS zones and hyperlink each Cloudera and hub VNETs to them.
The benefit of this strategy is that each knowledge warehouse and Liftie-based knowledge companies help pre-created personal DNS zones. There are nonetheless additionally a number of drawbacks:
- For Liftie, the personal DNS zone must be configured when registering the surroundings. As soon as previous the surroundings registration stage, it can’t be configured.
- DE will want a personal DNS zone for MySQL and it doesn’t help pre-configured personal DNS zones.
- On-premises networks can’t be linked to a personal DNS zone. If the DNS server is on an on-prem community, there aren’t any workable options.
Possibility 3: Create DNS Server as a Forwarder.
Create a few DNS servers (for HA consideration) with load balancer in Cloudera VNET, and configure conditional ahead to Azure Default DNS of the Cloudera VNET. Configure conditional ahead from the corporate customized personal DNS server to the DNS server within the Cloudera subnet.
The downside of this feature is that further DNS servers are required, which ends up in further administration overhead for the DNS staff.
Possibility 4: Azure-Managed DNS Resolve
Create a devoted /28 subnet in Cloudera VNET for Azure personal DNS resolver inbound endpoint. Configure conditional ahead from customized personal DNS to the Azure personal DNS resolver inbound endpoint.
Abstract
Bringing all issues collectively, take into account these greatest practices for organising your DNS with Cloudera on Azure:
- For the storage account, key vault, postgres DB
- Use service endpoints as the primary alternative.
- If service endpoint isn’t allowed, pre-create personal DNS zones and hyperlink to the VNET the place the DNS server is deployed. Configure conditional forwards from customized personal DNS to Azure default DNS.
- If the customized personal DNS is deployed within the on-premises community, use Azure DNS resolver or one other DNS server as DNS forwarder on the Cloudera VNET. Conditional ahead the DNS lookup from the personal DNS to the resolver endpoint.
- For the info warehouse, DataFlow, or machine studying knowledge companies
- Disable the personal DNS zone and use the general public DNS zone as an alternative.
- For the info engineering knowledge service
- Configure the Azure DNS resolver or one other DNS server as a DNS forwarder on the Cloudera VNET. Conditional ahead the DNS lookup from the personal DNS to the resolver endpoint. Please discuss with Microsoft documentation for the main points of organising an Azure DNS Non-public Resolver.
For extra background studying on community and DNS specifics for Azure, take a look at our documentation for the assorted knowledge companies: DataFlow, Knowledge Engineering, Knowledge Warehouse, and Machine Studying. We’re additionally completely happy to debate your particular wants; in that case please attain out to your Cloudera account supervisor or get in contact.