|
Utilizing Information Bases for Amazon Bedrock, basis fashions (FMs) and brokers can retrieve contextual info out of your firm’s non-public knowledge sources for Retrieval Augmented Era (RAG). RAG helps FMs ship extra related, correct, and customised responses.
Over the previous months, we’ve constantly added decisions of embedding fashions, vector shops, and FMs to Information Bases.
At present, I’m excited to share that along with Amazon Easy Storage Service (Amazon S3), now you can join your internet domains, Confluence, Salesforce, and SharePoint as knowledge sources to your RAG purposes (in preview).
New knowledge supply connectors for internet domains, Confluence, Salesforce, and SharePoint
By together with your internet domains, you can provide your RAG purposes entry to your public knowledge, reminiscent of your organization’s social media feeds, to reinforce the relevance, timeliness, and comprehensiveness of responses to person inputs. Utilizing the brand new connectors, now you can add your present firm knowledge sources in Confluence, Salesforce, and SharePoint to your RAG purposes.
Let me present you ways this works. Within the following examples, I’ll use the online crawler so as to add an internet area and join Confluence as an information supply to a information base. Connecting Salesforce and SharePoint as knowledge sources follows the same sample.
Add an internet area as an information supply
To provide it a strive, navigate to the Amazon Bedrock console and create a information base. Present the information base particulars, together with identify and outline, and create a brand new or use an present service position with the related AWS Identification and Entry Administration (IAM) permissions.
Then, select the information supply you need to use. I choose Net Crawler.
Within the subsequent step, I configure the online crawler. I enter a reputation and outline for the online crawler knowledge supply. Then, I outline the supply URLs. For this demo, I add the URL of my AWS Information Weblog writer web page that lists all my posts. You possibly can add as much as ten seed or start line URLs of the web sites you need to crawl.
Optionally, you possibly can configure customized encryption settings and the information deletion coverage that defines whether or not the vector retailer knowledge will likely be retained or deleted when the information supply is deleted. I maintain the default superior settings.
Within the sync scope part, you possibly can configure the extent of sync domains you need to use, the utmost variety of URLs to crawl per minute, and common expression patterns to incorporate or exclude sure URLs.
After you’re carried out with the online crawler knowledge supply configuration, full the information base setup by choosing an embeddings mannequin and configuring your vector retailer of selection. You possibly can verify the information base particulars after creation to watch the information supply sync standing. After the sync is full, you possibly can take a look at the information base and see FM responses with internet URLs as citations.
To create knowledge sources programmatically, you should use the AWS Command Line Interface (AWS CLI) or AWS SDKs. For code examples, try the Amazon Bedrock Person Information.
Join Confluence as an information supply
Now, let’s choose Confluence as an information supply within the information base setup.
To configure Confluence as an information supply, I present a reputation and outline for the information supply once more, and select the internet hosting methodology, and enter the Confluence URL.
To connect with Confluence, you possibly can select between base and OAuth 2.0 authentication. For this demo, I select Base authentication, which expects a person identify (your Confluence person account e-mail deal with) and password (Confluence API token). I retailer the related credentials in AWS Secrets and techniques Supervisor and select the key.
Observe: Ensure that the key identify begins with “AmazonBedrock-” and your IAM service position for Information Bases has permissions to entry this secret in Secrets and techniques Supervisor.
Within the metadata settings, you possibly can management the scope of content material you need to crawl utilizing common expression embody and exclude patterns and configure the content material chunking and parsing technique.
After you’re carried out with the Confluence knowledge supply configuration, full the information base setup by choosing an embeddings mannequin and configuring your vector retailer of selection.
You possibly can verify the information base particulars after creation to watch the information supply sync standing. After the sync is full, you possibly can take a look at the information base. For this demo, I’ve added some fictional assembly notes to my Confluence area. Let’s ask concerning the motion objects from one of many conferences!
For directions on find out how to join Salesforce and SharePoint as an information supply, try the Amazon Bedrock Person Information.
Issues to know
- Inclusion and exclusion filters – All knowledge sources help inclusion and exclusion filters so you possibly can have granular management over what knowledge is crawled from a given supply.
- Net Crawler – Keep in mind that you should solely use the online crawler by yourself internet pages or internet pages that you’ve got authorization to crawl.
Now out there
The brand new knowledge supply connectors can be found at this time in all AWS Areas the place Information Bases for Amazon Bedrock is out there. Verify the Area checklist for particulars and future updates. To study extra about Information Bases, go to the Amazon Bedrock product web page. For pricing particulars, overview the Amazon Bedrock pricing web page.
Give the brand new knowledge supply connectors a strive within the Amazon Bedrock console at this time, ship suggestions to AWS re:Put up for Amazon Bedrock or by means of your standard AWS contacts, and interact with the generative AI builder group at group.aws.
— Antje