Trieve’s Blog

Accurate Hallucination Detection With NER

Dens Sumesh — Tue, 07 Jan 2025 21:33:00 GMT

How We Do It: Smart Use of NER

Our method zeroes in on the most common and critical hallucinations—those that could mislead or confuse users. Based on our research, a large percentage of hallucinations fall into three categories:

Proper nouns (people, places, organizations)
Numerical values (dates, amounts, statistics)
Made-up terminology

Instead of throwing complex language models at the problem with a LLM-as-a-judge approach, we use Named Entity Recognition (NER) to spot proper nouns and compare them between the gen AI completion and the retrieved reference text. For numbers and unknown words, we use similarly straightforward techniques to flag potential issues.

Our approach will only work in use-cases where RAG is present which is fine given that Trieve is a search and RAG API. Further, because the most common approach to limiting hallucinations is RAG, this approach will work for any team building solutions on top of other search engines.

Why This Is Important:

Lightning fast: Processes in 100-300 milliseconds.
Fully self-contained: No need for external AI services.
Customizable: Works with domain-specific NER models.
Minimal setup: Can run on CPU nodes.

Benchmark Results

RAGTruth Dataset Performance

We achieved a 67% accuracy rate on the RAGTruth dataset, which provides a comprehensive benchmark for hallucination detection in RAG systems. This result is particularly impressive considering our lightweight approach compared to more complex solutions.

Comparison with Vectara

When tested against Vectara’s examples, our system showed:

70% alignment with Vectara’s model predictions
Comparable performance on obvious hallucinations
Strong detection of numerical inconsistencies
High accuracy on entity-based hallucinations

This level of alignment is significant because we achieve it without the computational overhead of a full language model.

Why This Works

Our method focuses on the types of hallucinations that matter most. Made-up entities, wrong numbers, and gibberish words. By sticking to these basics, we’ve built a system that:

Catches high-impact errors: No more fake organizations or incorrect stats.
Runs lightning fast: Minimal delay in real-time systems.
Fits anywhere: Easily integrates into production pipelines with no fancy hardware needed.

Why It Matters in the Real World

Speed and simplicity are the stars of this show. Our system processes responses in 100-300ms, making it perfect for:

Real-time applications (think chatbots and virtual assistants)
High-volume systems where efficiency is key
Low-resource setups, like edge devices or small servers

In short, this approach bridges the gap between effectiveness and practicality. You get solid hallucination detection without slowing everything down or breaking the bank.

What’s Next: Room to Grow

While we’re thrilled with these results, we’ve got a lot of ideas for the future:

Smarter Entity Recognition
- Train models for industry-specific jargon and custom entity types.
- Improve recognition for niche use cases.
Better Number Handling
- Add context-aware analysis for ranges, approximations, and units.
- Normalize and convert units for consistent comparisons.
Expanded Word Validation
- Incorporate specialized vocabularies for different fields.
- Make it multilingual and more context-aware.
Hybrid Methods
- Optionally tap into language models for tricky edge cases.
- Combine with semantic similarity scores or structural analysis for tougher challenges.

The Takeaway

Our system shows that you don’t need heavyweight tools to handle hallucination detection. By focusing on the most common issues, we’ve built a fast, reliable solution that’s production-ready and easy to scale.

It’s a practical tool for anyone looking to improve the trustworthiness of AI outputs, especially in environments where speed and resource efficiency are non-negotiable.

Check out our work, give it a try, and let us know what you think!

You can find all the code involved in our NER system, including benchmarks, at github.com/devflowinc/trieve/tree/main/hallucination-detection.

Uncle Flaviar: Conversational AI Drives 5-Figure Revenue Lift

Federico Chavez Torres — Mon, 19 May 2025 20:09:00 GMT

About Flaviar

Flaviar isn't just an e-commerce platform; it's the nation's—and quite possibly the world's—premier destination for wine and spirits. Since its founding in 2011 by Jugoslav Petkovic, Matija Rijavec, and Grisa Soba, Flaviar has become a true ecommerce unicorn, known for its innovation and success.

With a curated selection of rare and exclusive bottles, Flaviar offers members a unique journey of discovery through the world of fine beverages. Its innovative tasting boxes and personalized recommendations further elevate the experience, making it much more than just a place to buy drinks.

A Discerning AI for a Discerning Audience: How Trieve and Flaviar Teamed Up

Flaviar built its business on a deep understanding of its customers and products. Long before the current creator economy, they made an early bet on serving knowledgeable consumers who are constantly in the loop. This strategy has paid off, compounding over the years into a brand synonymous with reliability, quality, and a fiercely loyal community. When we connected with their team about integrating AI features, their forward-thinking culture was a perfect match for ours.

The challenge for Flaviar was clear: how to embrace the AI wave while maintaining their consistently on-brand, positive customer experience.

In Their Own Words

Vasja Fužir, Lead Product Manager at Flaviar, shares his perspective:

Trieve has genuinely transformed how our shoppers navigate the store. Their AI chat guides product discovery in a way that feels natural, intuitive, and genuinely helpful. From day one, their team has been incredibly responsive and proactive. They’ve acted like true partners, not just vendors—bringing ideas to the table, iterating fast, and helping us get real value quickly.

The tech lives up to its promise: personalized product suggestions, intelligent answers to shopper questions, and a smooth path from search to checkout. It adapts seamlessly to our catalog, syncs with our brand voice, and reduces the load on our support team—all without requiring heavy setup on our end.

This partnership ensures Flaviar remains at the forefront of AI innovation, with Trieve continuously delivering cutting-edge features that adapt to evolving market demands and customer expectations.

The Work & Results: Introducing Uncle Flaviar

We collaborated intensely with Flaviar to launch Uncle Flaviar, a sitewide, branded, and savvy AI concierge. Its sole purpose? To help customers discover the products they truly want—or better yet, the ones they didn't even know they needed.

Uncle Flaviar has consistently delivered a 5-figure monthly increase in revenue. We've observed that simple, effective "Quick Queries" like "Bottles to treat myself" or "Surprise me!" do wonders. They:

Create momentum in the purchasing journey.
Turn curiosity into purchase intent.
Convert purchase intent into add-to-carts with brief, compelling descriptions tailored to each product.

Beyond product discovery, Trieve's advanced capabilities mean it can ingest your specific policies, return rules, and other critical business logic. This allows the AI to accurately answer common "Where Is My Order?" (WISMO) and other customer service questions, further reducing support load and ensuring consistent, on-brand information delivery.

We're incredibly grateful to Flaviar and their customers for entrusting us with their queries. We remain committed to delivering the latest in enterprise-grade AI, as we've done for millions of searchers.

If you're looking to fill an AI-shaped gap in your business, we'd love to connect.

How to Build Agentic RAG for any PDF in 10 minutes

Nick Khami — Sun, 15 Jun 2025 21:01:00 GMT

Introduction

Retrieval Augmented Generation (RAG) has revolutionized how we build AI applications, allowing Large Language Models (LLMs) to answer questions based on custom data. But what if the LLM could decide when and how to search that data, like a smart assistant? That's where Agentic RAG comes in.

With Trieve, you can easily set up an agentic RAG pipeline that leverages advanced OCR for PDFs (via Chunkr) and gives your LLM the autonomy to intelligently query your knowledge base.

If you're not interested in the guide and just want to see the code in order to give it to your agent as a starting point, you can find a fully complete CLI demonstrating this functionality in a single file on github at github.com/devflowinc/trieve/blob/main/clients/cli/index.ts or install it via npm i -g trieve-cli.

Here's a short video where I break down how agentic RAG performs against Gemini with the entire file in the context window along with naive RAG (no agentic search) for the 2025 CrossFit Games Rulebook. Credit to canonical ai's original post.

Step 1: Sign Up for Trieve and Set Up Your Dataset

If you haven't already, sign up for a Trieve account at dashboard.trieve.ai. Once logged in, create a new dataset and upload your PDFs. Trieve will automatically process them using Chunkr, extracting text and metadata for efficient searching.

Prerequisites:

A Trieve account (sign up at dashboard.trieve.ai)
Copy your Trieve API_KEY, DATASET_ID, and ORGANIZATION_ID from the dashboard.
Node.js and npm/yarn installed.

Step 2: Initialize Your Node.js Project and Trieve Client

Create a new Node.js script (e.g., agentic-rag.js) and set up your Trieve client:

import fs from "fs";
import { TrieveSDK, UpdateDatasetReqPayload } from "trieve-ts-sdk";

// ---- Configuration ----
// Replace with your actual credentials
const TRIEVE_API_KEY = process.env.TRIEVE_API_KEY || "YOUR_TRIEVE_API_KEY";
const TRIEVE_DATASET_ID =
  process.env.TRIEVE_DATASET_ID || "YOUR_TRIEVE_DATASET_ID";
const TRIEVE_ORGANIZATION_ID =
  process.env.TRIEVE_ORGANIZATION_ID || "YOUR_TRIEVE_ORGANIZATION_ID";

if (
  TRIEVE_API_KEY === "YOUR_TRIEVE_API_KEY" ||
  TRIEVE_DATASET_ID === "YOUR_TRIEVE_DATASET_ID" ||
  TRIEVE_ORGANIZATION_ID === "YOUR_TRIEVE_ORGANIZATION_ID"
) {
  console.error(
    "Please set your TRIEVE_API_KEY, TRIEVE_DATASET_ID, and TRIEVE_ORGANIZATION_ID in the script or as environment variables.",
  );
  process.exit(1);
}

const trieveClient = new TrieveSDK({
  apiKey: TRIEVE_API_KEY,
  datasetId: TRIEVE_DATASET_ID,
  organizationId: TRIEVE_ORGANIZATION_ID, // Required for dataset updates
});

console.log("Trieve SDK initialized.");

Step 3: Configure Your Agent's Search Tool

For an LLM to act as an agent, it needs clear instructions on when and how to use its tools (in this case, searching your Trieve dataset). We configure this at the dataset level.

System Prompt (SYSTEM_PROMPT): This is the overarching instruction for the LLM. It should emphasize relying on the search tool.
Tool Description (tool_description): This tells the LLM when it should use the search tool. For robust RAG, you often want it to always use the tool.
Query Parameter Description (query_parameter_description): This guides the LLM on how to formulate its search queries effectively.

Tips and Tricks for Descriptions:

Be Explicit: Don't assume the LLM knows. Tell it directly, e.g., "ALWAYS call this search tool for EVERY user question."
Emphasize Data Freshness: Remind the LLM that its internal knowledge might be outdated and the search tool provides current information from your documents.
Encourage Specificity in Queries: Guide the LLM to extract keywords and form precise queries. Suggest trying multiple queries if the first attempt isn't fruitful.
Iterate: These descriptions are powerful. Experiment with different phrasings to see what works best for your use case.

Here's how to update your dataset configuration using the SDK:

async function configureSearchTool() {
  console.log("🔧 Configuring dataset for agentic search...");
  try {
    const updatePayload: UpdateDatasetReqPayload = {
      dataset_id: TRIEVE_DATASET_ID, // Ensure this is the correct dataset ID
      server_configuration: {
        SYSTEM_PROMPT: "You are an AI assistant that helps people find information in a set of documents. You have access to a search tool that can retrieve relevant information from the documents based on a query. YOU MUST ALWAYS CALL AND USE THE SEARCH TOOL FOR EVERY USER QUESTION WITHOUT EXCEPTION. Do not rely on your own knowledge - it may be outdated or incorrect. For each user question: 1) Use the search tool with a well-crafted query 2) If the first search doesn't yield satisfactory results, try additional searches with different terms 3) Only after searching should you formulate your response, citing the information found. Always inform the user that your answer is based on search results from their documents. If you don't find relevant information after multiple searches, be honest about this limitation.",
        TOOL_CONFIGURATION: {
          query_tool_options: {
            tool_description: "ALWAYS use the search tool for EVERY user question, even if you think you already know the answer. Your knowledge is limited and potentially outdated - you must rely on the provided search tool to get the most accurate and up-to-date information.",
            query_parameter_description: "Write a specific query with critical keywords from the user question. Use multiple search queries with different terms if needed to get comprehensive results.",
            // You can also define descriptions for other filters if you plan to use them agentically
            // price_filter_description: "The page range filter to use for the search",
            // max_price_option_description: "The maximum page to filter by",
            // min_price_option_description: "The minimum page to filter by",
          },
        },
      },
    };

    await trieveClient.updateDataset(updatePayload);
    console.log("✅ Dataset configuration updated successfully!");
  } catch (error) {
    console.error("❌ Failed to update dataset configuration:", error instanceof Error ? error.message : error);
  }
}

Step 4: Upload and Chunk Your PDF with Chunkr

Chunkr is Trieve's advanced file processing service. When you upload a file (like a PDF) with the chunkr_create_task_req_payload field, Chunkr uses sophisticated OCR technology that excels at understanding document layouts, tables, and images. This results in higher-quality chunks for your RAG pipeline.

Request Explanation:

base64_file: The file content, base64 encoded.
file_name: The original name of the file.
group_tracking_id (Optional but Recommended): A unique ID you can use to later check the status of the file processing and group related chunks. If not provided, Trieve might generate one.
chunkr_create_task_req_payload: {}: This empty object is the key! It signals Trieve to process this file using Chunkr for advanced chunking. You can pass specific Chunkr options here if needed, but an empty object uses sensible defaults.

async function uploadPdfWithChunkr(filePath: string, trackingId?: string) {
  console.log(`📤 Uploading PDF: ${filePath} with Chunkr...`);
  try {
    const fileBuffer = fs.readFileSync(filePath);
    const base64File = fileBuffer.toString('base64');
    const fileName = filePath.split('/').pop() || filePath;

    const generatedTrackingId = trackingId || `chunkr-doc-${fileName}-${Date.now()}`;

    const response = await trieveClient.uploadFile({
      base64_file: base64File,
      file_name: fileName,
      group_tracking_id: generatedTrackingId,
      chunkr_create_task_req_payload: {}, // This enables Chunkr processing!
    });

    console.log("📄 File upload initiated. Response:", response);
    console.log(`✨ File sent to Chunkr for processing. Tracking ID: ${generatedTrackingId}`);
    console.log("⏳ You can check processing status using this tracking ID with other Trieve endpoints.");
    return generatedTrackingId; // Return for potential status checking
  } catch (error) {
    console.error("❌ PDF upload failed:", error instanceof Error ? error.message : error);
  }
}

Note: File processing with Chunkr is asynchronous. The uploadFile endpoint returns quickly, but the actual chunking happens in the background. You'd typically use the group_tracking_id to poll for completion status if needed, though for this example, we'll assume it completes.

Step 4: Asking Agentic Questions

Now for the magic! To ask an agentic question:

Create a Topic: Topics are like conversation threads. Each agentic interaction typically happens within a topic.
Create a Message Reader: When creating a message within a topic, set:
- use_agentic_search: true: This tells Trieve to use the agentic flow defined by your dataset's tool configuration.
- model: Specify an LLM capable of agentic behavior. Trieve offers models like o3 (Claude 3 Opus), c3.5s (Claude 3.5 Sonnet), gpro (GPT-4o), gpt4t (GPT-4 Turbo) which are well-suited for this. Check Trieve documentation for the latest list of supported models. o3 is a powerful option.

The response will be streamed back, often including "thinking" steps from the agent before the final answer.

import { TrieveSDK, ChunkMetadata, Topic } from 'trieve-ts-sdk';
import chalk from 'chalk'; // Optional: for styled console output

// Assume trieveClient is already initialized and configured globally for this script
// const trieveClient = new TrieveSDK({ ... });

async function askAgenticQuestion(
  question: string,
  existingTopicId?: string,
  userId: string = 'default-blog-user-' + Date.now() // Example user ID
): Promise<{ actualAnswer: string; parsedChunks: ChunkMetadata[]; topicId: string } | undefined> {
  console.log(chalk.blue(`🤔 Asking agentic question: "${question}"`));
  try {
    let topicIdToUse = existingTopicId;

    if (!topicIdToUse) {
      const topicName = question.substring(0, 50) + "..."; // Simple topic name
      console.log(chalk.magenta(`📝 Creating new topic: "${topicName}" for user "${userId}"`));
      // Ensure Topic type is correctly imported if using createTopic's return type explicitly
      const topicData: Topic = await trieveClient.createTopic({
        name: topicName,
        owner_id: userId,
      });
      topicIdToUse = topicData.id;
      console.log(chalk.magenta(`🏷️ Topic created with ID: ${topicIdToUse}`));
    }

    // This check is good practice, though topicIdToUse should be set if creation was successful
    if (!topicIdToUse) {
        console.error(chalk.red("❌ Critical error: Topic ID could not be established."));
        return undefined;
    }

    const { reader } = await trieveClient.createMessageReaderWithQueryId({
      topic_id: topicIdToUse,
      new_message_content: question,
      use_agentic_search: true, // Crucial for agentic behavior
      model: "o3", // Or "c3.5s", "gpro", "gpt4t". Using a powerful model is recommended.
                    // "o3" refers to Claude 3 Opus in Trieve's context.
    });

    console.log(chalk.cyan("💬 Streaming response from agent:"));
    process.stdout.write(chalk.bold("🤖 Agent: ")); // Start the agent's response line

    const decoder = new TextDecoder();
    let actualAnswer: string = '';
    let parsedChunks: ChunkMetadata[] = [];
    let chunkDataAccumulator: string = ''; // Accumulates parts of the stream that might form chunk JSON
    let isParsingChunks: boolean = false;   // Flag: actively accumulating/expecting chunk JSON
    let isThinkingSection: boolean = false; // Flag: agent is sending "thinking" status messages

    while (true) {
      const { done, value } = await reader.read();
      if (done) break;

      const streamChunkText = decoder.decode(value);

      // 1. Handle "thinking" or status messages from the agent stream
      if (streamChunkText.includes('🤔') || streamChunkText.includes('📝') || streamChunkText.includes('✅') || streamChunkText.includes('🔍')) {
        isThinkingSection = true; // We are in a special status update from the agent
        process.stdout.write(chalk.yellow(streamChunkText)); // Print thinking messages
        continue; // Process next part of the stream
      } else {
        // If we were in a thinking section and now receive a non-thinking chunk, reset the flag.
        // This ensures subsequent text is treated as part of the answer or chunk data.
        isThinkingSection = false;
      }

      // 2. Detect the start of the chunk JSON data section (inspired by CLI's logic)
      // Assumes chunk data starts with '[{'
      if (streamChunkText.includes('[{') && !isParsingChunks) {
        isParsingChunks = true;
        chunkDataAccumulator = ''; // Reset accumulator for this new potential JSON block
      }

      // 3. Accumulate and parse chunk JSON data
      if (isParsingChunks) {
        chunkDataAccumulator += streamChunkText;

        // Check for the delimiter "||" which, by CLI convention, separates chunk JSON from the main answer
        if (chunkDataAccumulator.includes('||')) {
          isParsingChunks = false; // We've likely received the full chunk JSON block
          const parts = chunkDataAccumulator.split('||');
          const jsonDataPart = parts[0].trim();

          if (jsonDataPart) {
            try {
              // The CLI expects the stream to format chunks as: [{ "chunk": ChunkMetadata }, ...]
              const rawChunkObjects: { chunk: ChunkMetadata }[] = JSON.parse(jsonDataPart);
              if (Array.isArray(rawChunkObjects)) {
                parsedChunks = rawChunkObjects.map(item => item.chunk);
                // Optional: Log that chunks were found during streaming
                process.stdout.write(chalk.dim(`\n[ℹ️ System: Extracted ${parsedChunks.length} reference chunks.]\n`));
                process.stdout.write(chalk.bold("🤖 Agent: ")); // Re-prompt for agent answer part
              }
            } catch (e) {
              process.stdout.write(chalk.yellow('\n⚠️ Warning: Could not parse chunk JSON. Content (partial): ' + jsonDataPart.substring(0, 100) + "...\n"));
              // If parsing fails, it might be an error or unexpected format.
              // The CLI logs an error. Consider if this data should be part of `actualAnswer`.
              // For now, mirroring CLI's behavior of just warning.
            }
          }

          // The part after "||" is considered the start of the textual answer from the LLM
          if (parts[1]) {
            const answerInitialPart = parts[1];
            actualAnswer += answerInitialPart;
            process.stdout.write(answerInitialPart);
          }
          chunkDataAccumulator = ''; // Clear the accumulator
        }
        // If "||" is not yet found, `chunkDataAccumulator` continues to build in next iteration.
      } else if (!isThinkingSection) {
        // 4. Accumulate the actual textual answer from the LLM
        // This runs if a) not parsing chunks, and b) not in a "thinking" message from the agent
        actualAnswer += streamChunkText;
        process.stdout.write(streamChunkText);
      }
    }

    reader.releaseLock();
    process.stdout.write('\n'); // Ensure a new line after the full streamed response

    console.log(chalk.green('\n✅ Agentic response complete.'));

    // Optional: Summarize found chunks after the stream
    if (parsedChunks.length > 0) {
        console.log(chalk.blueBright(`\n📚 Summary of ${parsedChunks.length} references found:`));
        parsedChunks.forEach((chunk, index) => {
            console.log(
                chalk.grey(`  Ref ${index + 1}: ID (${chunk.tracking_id || chunk.id.substring(0,8)}) `) +
                (chunk.link ? chalk.cyan(`Link: ${chunk.link} `) : '') +
                (chunk.metadata ? chalk.magenta(`File: ${chunk.metadata.file_name || 'N/A'}`) : '')
            );
        });
    } else {
        console.log(chalk.yellow('⚠️ No reference chunks were explicitly parsed from this stream via "||" delimiter.'));
    }

    return { actualAnswer, parsedChunks, topicId: topicIdToUse };

  } catch (error) {
    console.error(chalk.red('❌ Failed to process agentic question:'), error instanceof Error ? error.message : error);
    return undefined;
  }
}

Putting It All Together

Let's create a simple main function to run these steps. Make sure you have a PDF file (e.g., sample.pdf) in the same directory or provide a full path.

async function main() {
  // Step 1: Configure the dataset's search tool (run once or when you want to update)
  await configureSearchTool();

  // Step 2: Upload a PDF using Chunkr
  // Replace 'sample.pdf' with the path to your PDF file
  const pdfFilePath = "sample.pdf";
  if (!fs.existsSync(pdfFilePath)) {
    console.error(
      `Error: PDF file not found at ${pdfFilePath}. Please create a sample.pdf or update the path.`,
    );
    return;
  }
  const trackingId = await uploadPdfWithChunkr(pdfFilePath);

  if (!trackingId) {
    console.error("File upload failed, cannot proceed to ask question.");
    return;
  }

  // Give some time for Chunkr to process (in a real app, you'd poll status)
  console.log(
    "\n⏳ Waiting 30 seconds for Chunkr to process the PDF (adjust as needed for larger files)...",
  );
  await new Promise((resolve) => setTimeout(resolve, 30000));

  // Step 3: Ask an agentic question related to the content of your PDF
  const question = "What are the main topics discussed in this document?"; // Change this to fit your PDF
  const result = await askAgenticQuestion(question);

  if (result) {
    console.log(`\n🗣️ You asked: "${question}"`);
    // The fullResponse might contain structured data depending on the stream.
    // For this example, we primarily focused on printing text as it came.
    // console.log(`\n📝 Agent's Full Streamed Response (may include intermediate steps):\n${result.fullResponse}`);
  }
}

main().catch(console.error);

To run this:

Save the combined code as agentic-rag.js.
Place a sample.pdf in the same directory or update pdfFilePath.
Set your environment variables or update the placeholders in the script.
Run node agentic-rag.js (if you're not using TypeScript directly, you might need ts-node agentic-rag.ts or compile it first).

Conclusion

You've just built a powerful Agentic RAG pipeline! By configuring your Trieve dataset's tool descriptions, leveraging Chunkr for superior PDF processing, and enabling agentic search, you've empowered an LLM to intelligently query your custom documents.

This is just the beginning. You can expand on this by:

Implementing robust status checking for file uploads.
Building a more sophisticated UI to handle streamed responses and citations.
Experimenting with different agentic models and prompt engineering for your tool descriptions.

Trieve makes complex AI tasks like Agentic RAG surprisingly accessible. Happy building!

Build a Hotel Voice Agent with Trieve + Vapi

Tue, 04 Mar 2025 18:12:00 GMT

If you are a more API focused user and just interested in the HTTP requests you would use to build this system, then please reference the included cURL requests throughout the blog.

Hotel Voice Agent Requirements

We are going to be building a voice assistant for Hotel Spero in San Francisco.

[x] Answer questions about check-in/check-out time
[x] Provide directions to the hotel and parking information
[x] Provide information about hotel amenities (e.g., gym, pool, restaurant, spa)

Trieve Portion

Creating a Trieve account is a pre-requisite to this tutorial. Navigate to trieve.ai and click "Sign Up" to create your account and access the dashboard. Alternatively, navigate directly to dashboard.trieve.ai.

Step 1: Create a Trieve Dataset

Datasets in Trieve correlate to knowledge bases. Typically you will want to have one dataset per voice agent in your system.

You can create your first dataset by clicking on the "create dataset" button when you sign up for Trieve:

After pressing that button you will see the following modal, simply enter your dataset name and press enter. You can ignore the advanced settings.

You can also do this via an HTTP request. If creating the dataset via HTTP, make sure to copy the ID of the dataset from the response and save it for later requests.

curl 'https://api.trieve.ai/api/dataset' \
  -H 'Authorization: <replace-with-your-trieve-api-key>' \
  -H 'cache-control: no-cache' \
  -H 'content-type: application/json' \
  -H 'tr-organization: <replace-with-your-organization-id>' \
  --data-raw '{"dataset_name":"vignette-san-francisco"}'

Step 2: Trigger a crawl

Hotel Spero's website contains information to answer every common question about the hotel. Parking, directions, gym, pool, and other details are all available. We will be performing a crawl of their website to make that information accessible to our voice agent.

Navigate to the Crawling Options in the left sidenav on the dashboard page for your dataset
Paste the site's URL into the Site URL input
Click start new crawl

You can see the progress in the results page after your crawl is triggered:

You can also trigger the crawl through an HTTP request:

curl 'https://api.trieve.ai/api/crawl' \
  -H 'Authorization: <replace-with-your-trieve-api-key>' \
  -H 'content-type: application/json' \
  -H 'tr-dataset: <replace-with-your-dataset-id>' \
  --data-raw '{"crawl_options":{"allow_external_links":false,"boost_titles":true,"exclude_paths":[],"exclude_tags":["navbar","footer","aside","nav","form", "header"],"include_paths":[],"include_tags":[],"interval":"daily","limit":1000,"site_url":"https://www.ihg.com/vignettecollection/hotels/us/en/san-francisco/sfosh/hoteldetail","scrape_options":null}}'

Vapi Portion

Now that we have all of the Hotel's context pulled into to a Dataset, we will need to create a Voice assistant with Vapi and connect it to the Trieve Dataset for Hotel Spiro as a Knowledge Base.

<Info> Creating a Vapi account is a pre-requisite to this tutorial. Navigate to vapi.ai and click "Sign Up" to create your account and access the dashboard. Alternatively, navigate directly to dashboard.vapi.ai. </Info>

Add your Trieve API Key as a Vapi Provider Credential

Create and copy Trieve api key in your Trieve dashboard at dashboard.trieve.ai

Click on the on the create api key button in the "API Key" tab of the Trieve dashboard.

Copy your API key from Trieve

You will need to copy the key to add it to Vapi after generating it.

Navigate your Provider Credentials Page in Vapi

Now that you have the Trieve copied, you need to navigate to the provider credentials settings in Vapi.

Add your Trieve API key to Vapi

Vapi needs to have your Trieve key so it can make API calls to your Dataset within Trieve.

Create a Vapi Knowledge Base that connects to Trieve

Vapi has a HTTP API route documented at docs.vapi.ai/api-reference/knowledge-bases/create which allows you to create a knowledge base.

Save your Dataset ID from the Trieve dashboard

Open your dataset in the Trieve dashboard and save it's ID so you can connect it to Vapi via HTTP API call later. You will need to copy this later.

Copy your Vapi secret key from the Vapi dashboard

You will need your secret API key from Vapi in order to complete the final required step and connect your Trieve knowledge-base.

Make the HTTP API call to create the Knowledge Base in Vapi that connects to Trieve

Navigate to Vapi's docs for the request at docs.vapi.ai/api-reference/knowledge-bases/create and click on the "try it" button. Then follow these steps.

Once the popover opens, you will need to fill in the fields to make the request in order to create your knowledge base.

Paste your Vapi api key into the token text input field
Select Trieve for the request body
Select Import for createPlan
Paste your Trieve Dataset ID for the providerId
Add the name parameter and set it to the name of your KB
Press send request

If your request succeeds, you will see the following message:

This step can also be performed via a HTTP request through cURL or another method of your choice:

curl -X POST https://api.vapi.ai/knowledge-base \
     -H "Authorization: Bearer <replace-with-your-vapi-key>" \
     -H "Content-Type: application/json" \
     -d '{
      "provider": "trieve",
      "searchPlan": {
        "searchType": "hybrid",
        "scoreThreshold": 0
      },
      "createPlan": {
        "type": "import",
        "providerId": "<replace-with-your-trieve-api-key>"
      },
      "name": "IHG-SF-Hotel"
    }'

Setup your Voice agent in the Vapi dashboard

The final step required to create your hotel voice agent is creating the Assistant inside of Vapi.

Click Create Assistant
Setup your prompts
Select the knowledge base you named and created in the previous step.

Here is the raw text of our first message and prompts:

First message: Hi! I\'m James here to assist you with any questions about the Spiro hotel in San Francisco

System prompt: You are Steve, the friendly and witty voice assistant for Hotel Spiro located in the heart of downtown San Francisco. Our doors are open 24/7 to welcome guests from near and far. You handle inquiries about the hotel, from room availability and amenities to event bookings.  Here’s your conversational blueprint:  1\. Start with a lighthearted greeting. 2\. Gather the caller\'s full name and purpose for calling. 3\. Use the retrieved context to help yourself respond to users and assist them with their queries about the hotel    Remember to keep things casual and quick, like a friendly chat. Be concise, using friendly fillers like \"Well...\", \"You know...\", and \"Let\'s see...\". Your goal is to make guests feel welcome while efficiently helping with their requests.

If you would prefer, you can also do this via HTTP request as follows:

curl -X POST https://api.vapi.ai/assistant \
     -H "Authorization: Bearer bec788fe-cdaf-4126-82ee-a90aa06dad47" \
     -H "Content-Type: application/json" \
     -d '{
      "model": {
        "provider": "openai",
        "model": "gpt-4o-mini",
        "knowledgeBaseId": "31bd355c-1137-4a1f-93a9-f01020207d05",
        "messages": [
          {
            "content": "You are Steve, the friendly and witty voice assistant for Hotel Spiro located in the heart of downtown San Francisco. Our doors are open 24/7 to welcome guests from near and far. You handle inquiries about the hotel, from room availability and amenities to event bookings.  Here’s your conversational blueprint:  1\. Start with a lighthearted greeting. 2\. Gather the caller\'s full name and purpose for calling. 3\. Use the retrieved context to help yourself respond to users and assist them with their queries about the hotel    Remember to keep things casual and quick, like a friendly chat. Be concise, using friendly fillers like \"Well...\", \"You know...\", and \"Let\'s see...\". Your goal is to make guests feel welcome while efficiently helping with their requests.",
            "role": "system"
          }
        ]
      },
      "firstMessage": "Hi! I\'m James here to assist you with any questions about the Spiro hotel in San Francisco",
      "name": "James"
    }'

Conclusion

And with that, you've successfully built a hotel voice assistant using Trieve and Vapi! By leveraging Trieve's powerful knowledge base capabilities and Vapi's voice agent platform, you've created a system that can answer common customer questions, provide directions, and offer information about hotel amenities. This setup not only enhances the guest experience but also streamlines operations, freeing up staff to focus on more complex tasks.

This tutorial provides a solid foundation for building similar voice agents for other hotels or even different types of businesses. Remember to tailor your datasets and prompts to the specific needs of your target audience. As you continue to explore the possibilities of AI-powered voice assistants, consider experimenting with different AI models, prompt engineering techniques, and knowledge base strategies to further optimize your results. The future of customer service is conversational, and with tools like Trieve and Vapi, you're well-equipped to be at the forefront of this exciting revolution. Happy building!

How we Built 300μs Typo Detection for 1.3M Words in Rust

Dens Sumesh — Mon, 09 Sep 2024 13:42:00 GMT

We launched our Hacker News search and RAG engine with a half-baked typo correction system. Our first draft took 30+ms for correctly spelled queries which was slow enough that we defaulted it to off. Our latest version is 100 times faster, 300μs for correctly spelled queries and ~5ms/word for misspellings. We explain how it was accomplished in this post!

![video demo of spellcheck](https://cdn.trieve.ai/blog/building-30%CE%BCs-typo-tolerance-for-1.3M-words%20using%20Rust/typo-tolerance-demo.gif)

Sample Queries You Can Try

Click the links to try the typo correction system out yourself on hn.trieve.ai.

Creating a dictionary of Words and Frequencies

For small datasets, this is an easy task. You can scroll ~1000 HN post size text blobs in 10 seconds with one worker and basic word splitting. However, as you scale to the size of our Hacker News Demo (38M+ posts), work needs to be distributed.

Eventually, we decided on 2 distinct workers for dictionary building:

Cronjob to scroll all of the documents present in each of our users' search indices and add chunk ids from our database into a Redis queue 500 at a time.
Word worker that pops off the queue and procesesses 500 chunks at a time. Text for each chunk is pulled, split into words, and each word is then loaded into Clickhouse.

We chose ClickHouse to store the dictionary as we ran into deadlock and performance issues with Postgres writes as we scaled the number of workers. ClickHouse's async inserts are fantastic for this task and allowed us to ingest the entire 38M+ document dataset in < 1hr.

Using a BKTree data structure to identify and correct typos

We take the standard approach to typo correction and build per-dataset Burkhard-Keller Trees (BKTrees) for efficient comparision of words in the search query and the dataset's dictionary in O(log N) time complexity. Explaining this data structure in depth is outside the scope of this blog, but you can read our Rust implementation here or its wiki for more information.

We utilized a third bktree-worker to build the BKTrees. It takes datasets with completed dictonaries stored in Clickhouse then uses their words and frequencies to construct a tree.

Once the BKTree is constructed, the worker then stores it in Redis such that it can be efficiently loaded into the API server's memory when needed at first query time for a given dataset.

This was challenging for larger datasets where the tree was hundreds of megabytes large and timed out redis on write and read. We developed a serialization method which flattens and gzips to reduce the size in redis as well as the latency when pulling and pushing from it.

Writing the Business Logic to Perform Typo Corrections

On the API server side, in our typo_operator, we optimized to reduce time required for corrections down to ~300μs for correctly spelled queries and ~10ms/word for mispelled queries.

![typo-operator-graph](https://cdn.trieve.ai/blog/building-30%CE%BCs-typo-tolerance-for-1.3M-words%20using%20Rust/typo-operator-graph.webp)

Pulling from Redis

Pulling a massive data structure, like the BKTree for HN, from Redis takes 300+μs. This is nonviable to do on each search, so we developed a cache layer server-side to store BKTrees after they had been pulled once using lazy_static!

lazy_static! {
    static ref BKTREE_CACHE: BKTreeCache = BKTreeCache::new();
}

On the first search with typo-tolerance enabled, we initiate a ~200-400ms cold start to pull the BKTree for the dataset being queried from Redis into server memory. Searches following this operation then use the BKTree to check for typos which only takes 100-300μs.

Identifying English Words

1. Preliminary English Word Identification

Since our BKTrees are constructed solely from dataset-specific dictionaries, they may not have all valid English words. To prevent inaccurate corrections of legitimate words absent from our trees, we use a preliminary English word identification step:

We maintain an in-memory hashset of approximately 400,000 English words, stored using lazy_static!.

static ref ENGLISH_WORDS: HashSet<String> = {
        include_str!("../words.txt")
            .lines()
            .map(|s| s.to_lowercase())
            .collect()
    };

2. Affix Analysis

We then check for if the word is just an english word with a prefix or suffix:

We construct separate Tries for common prefixes and suffixes.

static ref PREFIX_TRIE: Trie = {
        let prefixes = vec![
            "anti", "auto", "de", "dis", "down", "extra", "hyper", "il", "im", "in", "ir", "inter",
            "mega", "mid", "mis", "non", "over", "out", "post", "pre", "pro", "re", "semi", "sub",
            "super", "tele", "trans", "ultra", "un", "under", "up",
        ];
        Trie::new(&prefixes)
    };
    static ref SUFFIX_TRIE: Trie = {
        let suffixes = vec![
            "able", "al", "ance", "ation", "ative", "ed", "en", "ence", "ent", "er", "es", "est",
            "ful", "ian", "ible", "ic", "ing", "ion", "ious", "ise", "ish", "ism", "ist", "ity",
            "ive", "ize", "less", "ly", "ment", "ness", "or", "ous", "s", "sion", "tion", "ty",
            "y",
        ];
        Trie::new(&suffixes)
    };

For each word in the query, we search these Tries to identify the longest matching prefix and suffix.
We then strip these affixes from the word, leaving us with the root.
After stripping, we perform a final dictionary check:
The stripped root is searched against the english word corpus.

fn is_likely_english_word(word: &str) -> bool {
    if ENGLISH_WORDS.contains(&word.to_lowercase()) {
        return true;
    }

    // Check for prefix
    if let Some(prefix_len) = PREFIX_TRIE.longest_prefix(word) {
        if ENGLISH_WORDS.contains(&word[prefix_len..].to_lowercase()) {
            return true;
        }
    }

    // Check for suffix
    if let Some(suffix_len) = SUFFIX_TRIE.longest_suffix(word) {
        if ENGLISH_WORDS.contains(&word[..word.len() - suffix_len].to_lowercase()) {
            return true;
        }
    }

    // Check for compound words
    if word.contains('-') {
        let parts: Vec<&str> = word.split('-').collect();
        if parts
            .iter()
            .all(|part| ENGLISH_WORDS.contains(∂.to_lowercase()))
        {
            return true;
        }
    }

    false
}

4. BKTree Search for Non-Dictionary Words

For words that don't pass our English word checks, we initiate a BKTree search:

Query the BKTree to find the closest matching words.
Generate a set of candidate corrections for each non-dictionary word.

let mut best_correction = None;
            let mut best_score = 0;

            for ((correction, freq), distance) in tree.find(word.to_string(), max_distance) {
                if distance == 0 {
                    best_correction = None;
                    break;
                }
                if !is_best_correction(word, correction) {
                    continue;
                }

                let score = (max_distance - distance) * 1000 + *freq as isize;

                if score > best_score || best_correction.is_none() {
                    best_correction = Some(correction);
                    best_score = score;
                }
            }

            if let Some(correction) = best_correction {
                corrections.insert(word, correction.to_string());
            }

5. Correction Selection

From the set of correction candidates, we use a scoring algorithm to select the best correction:

Our algorithm prioritizes prefix matches and factors in the frequency of each candidate word within the dataset.

fn is_best_correction(word: &str, correction: &str) -> bool {
    // Length-based filter
    let len_diff = (word.len() as i32 - correction.len() as i32).abs();
    if len_diff > 2 {
        return false;
    }

    // Prefix matching (adjust the length as needed)
    let prefix_len = std::cmp::min(1, std::cmp::min(word.len(), correction.len()));
    if word[..prefix_len] != correction[..prefix_len] {
        return false;
    }

    // Character set comparison
    let word_chars: HashSet<char> = word.chars().collect();
    let correction_chars: HashSet<char> = correction.chars().collect();
    let common_chars = word_chars.intersection(&correction_chars).count();
    let similarity_ratio =
        common_chars as f32 / word_chars.len().max(correction_chars.len()) as f32;

    similarity_ratio >= 0.8
}

Unexpected Benefits

Levitating commented on HN that a query for FreeBSD sorted by points returned irrelevant results. Our tokenizer splits on camel case so FreeBSD got turned into Free BSD FreeBSD and stories only containing the word "Free" had more points than anything containing "FreeBSD" and thus ranked higher. Being able to effectively check for full words on a dataset-level allowed us to automatically require non-English words such that a query for FreeBSD turned into "FreeBSD".

![auto required word demo](https://cdn.trieve.ai/blog/building-30%CE%BCs-typo-tolerance-for-1.3M-words%20using%20Rust/auto-required-word-demo.gif)

Future Ideas

We plan to leverage this same system to implement query splitting and concatenation as those features share the same requirement of quickly looking up words in a dictionary.

Trieve will always pursue the best possible relevance out of the box! Try it on our HN search engine, sign up for a free cloud account, or see our self-hosting guides.

Building Search For the YC Company Directory With Trieve, Bun, and SolidJS

Nick Khami — Mon, 19 Feb 2024 18:52:00 GMT

Introduction

Want to sign on Trieve as a Vendor or just talk to a human? Book a meeting with a founder at Trieve here on cal.com. Also star us on Github here if you have not already!

This blog post is going to act as a tutorial showing you how to use Trieve in order to build updated search for the YC company directory using bun and SolidJS.

See the complete code for this tutorial at github.com/devflowinc/yc-companies and try the finished project at yc.trieve.ai.

Step 1. Exploratory Data Analysis

Before doing anything else, we need to figure out how we are going to scrape the data in order to build this demo. The first step of that is looking at the public YC company directory and seeing what public data is available.

1. Each listed company is represented via its own page linked in the search result list

Turns out that there is a lot you can learn from inspect element. In this case by looking at the DOM and having the context that a lot of YC software development is in Rails, I am willing to gamble that there is a "view" for each company. This is because Rails traditionally follows a MVC (Model View Controller) pattern.

![Company page link available on each listing](https://cdn.trieve.ai/yc-directory-tutorial/yc-dir-company-links.png)

2. The company view offers the data in a neat JSON object as an HTML attribute!

Aha! I feel a sense of vindication looking at the DOM for AirBNB as it does seem to be a traditional view. Even luckier, all of the data about a given company is present in an attribute with in the first element of the page.

![JSON data representing company available in DOM attribute](https://cdn.trieve.ai/yc-directory-tutorial/yc-dom-attribute-with-data.png)

3. Plan out an approach for pulling this JSON object from the DOM for every company

We have discovered that each YC company in the directory has a page with a JSON object containing all of its data, now what?

1. Write a script to get the link to every company available in the directory

I don't think it's too relevant to this tutorial to explain how I did this, but here is a a link to the full github gist containing the script I pasted into the console to get all of the company links if you are interested in reading it. My co-founder Denzell always gets a laugh over how much work I'm willing to do in the console.

4. Settle on a language for pulling each company's view from the link and getting the JSON object

Typically in these spots I pick python because beautifulsoup4 is such an amazing package for these kinds of things done headlessly on a server. However, I am more comfortable in JS and vaguely knew that bun, a new javascript server runtime written in Zig, was advertised as a good solution as well.

Because I wanted to try Bun out and this should be a relatively small task either way, I opted for typescript ran via bun.

Step 2. Sign up For Trieve's Free Cloud To Get an API_KEY and DATASET_ID

Navigate to dashboard.trieve.ai and sign in or make an account
On the first page you see, click create dataset
On the dataset creation page, save your dataset_id to use later
Click the button to create an API key
Create a Read+Write type API key, save the value to use later on

Step 3. Use bun to pull the DOM containing the view for each company and extract the data from it

1. Install bun and scaffold an empty project using `bun init`

I am going to cede to the bun docs for bun init here instead of detailing how this works. On the bun docs page, you can also find an install guide.

2. Use the `happy-dom` package to headlessly pull data from the View for each company

To start, run bun add -d @happy-dom/global-registrator to add happy-dom as a dependency to your bun project.

Now, at the top of your index.ts file we are going to import Window from happy-dom and instantiate a document to use querySelector with later on.

Note that we add the docstring comment for the dom lib such that we get the benefit of typescript types for the DOM objects.

/// <reference lib="dom" />

import { Window } from "happy-dom";

const window = new Window();
const document = window.document;

Continuing, we are going to write a function to make a GET request for the HTML page at each company's URL and extract its data.

const processLink = async (companyUrl: string, groupIds: string[]) => {
  try {
    const pageRespHtml = await fetch(companyUrl);
    const pageRespText = await pageRespHtml.text();
    document.body.innerHTML = pageRespText;

    // get the first div that has a data-page attribute
    const divs = document.body.querySelectorAll("div");
    divs.forEach(async (div) => {
      const dataPage = div?.getAttribute("data-page");
      if (!dataPage) {
        return;
      }

      const bulkData = JSON.parse(dataPage).props;
      const companyData = bulkData.company;
      await processCompanyChunk(companyData, groupIds);
    });
  } catch (e) {
    console.error("error processing link", companyUrl, e);
    return;
  }
};

3. Stitch each companies structured data into an unstructured text blob and send it to Trieve for indexing and storage

First, because Trieve does not yet have a typescript SDK, we will specify the request data for creating a chunk as an interface.

export interface CreateChunkData {
  chunk_html: string;
  group_ids: string[];
  link: string;
  tag_set: string[];
  tracking_id: string;
  metadata: Object;
}

Next, we will write a function that accepts a CreateChunkData type object and then uses it to call Trieve's create_chunk route.

const createChunk = async (chunkData: CreateChunkData) => {
  const response = await fetch(`${API_URL}/chunk`, {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      Authorization: Bun.env.API_KEY ?? "",
      "TR-Dataset": Bun.env.DATASET_ID ?? "",
    },
    body: JSON.stringify(chunkData),
  });
  if (!response.ok) {
    console.error("error creating chunk", response.status, response.statusText);
    const respText = await response.text();
    console.error("error creating chunk", respText);
    return "";
  }

  const responseJson = await response.json();
  if (!response.ok) {
    console.error("error creating chunk", responseJson.message);
    return "";
  }
  console.log("success creating chunk", responseJson.chunk_metadata.id);
  const chunkId = responseJson.chunk_metadata.id;

  return chunkId as string;
};

To fully process the data we still need to stitch each company's structured data into an unstructured blob which we can use for the chunk_html.

const processCompanyChunk = async (
  bulkDataCompany: any,
  groupIds: string[],
) => {
  const group_id = await createChunkGroup(
    bulkDataCompany.name as string,
    bulkDataCompany.one_liner as string,
  );

  const companyName = "<h1>" + bulkDataCompany.name + "</h1>";
  const companyOneLiner = "<h3>" + bulkDataCompany.one_liner + "</h3>";
  const companyLongDescription =
    "<p>" + bulkDataCompany.long_description + "</p>";
  const companyLocation =
    "<p>" +
    "Located in " +
    bulkDataCompany.location +
    ", " +
    bulkDataCompany.country +
    " and founded in " +
    bulkDataCompany.year_founded +
    "</p>";
  const chunk_html =
    "<div>" +
    companyName +
    companyOneLiner +
    companyLongDescription +
    companyLocation +
    "</div>";

  const link = bulkDataCompany.ycdc_company_url;

  const tag_set =
    bulkDataCompany.batch_name +
    "," +
    bulkDataCompany.tags.join(",") +
    "," +
    bulkDataCompany.city_tag;

  const tracking_id = bulkDataCompany.id.toString();

  const company_name = bulkDataCompany.name;
  const company_one_liner = bulkDataCompany.one_liner;
  const company_long_description = bulkDataCompany.long_description;
  const batch = bulkDataCompany.batch_name;
  const company_location = bulkDataCompany.location;
  const company_city = bulkDataCompany.city;
  const company_city_tag = bulkDataCompany.city_tag;
  const company_country = bulkDataCompany.country;
  const company_year_founded = bulkDataCompany.year_founded;
  const company_website = bulkDataCompany.website;
  const company_linkedin = bulkDataCompany.linkedin_url;
  const company_twitter = bulkDataCompany.twitter_url;
  const company_facebook = bulkDataCompany.facebook_url;
  const company_crunchbase = bulkDataCompany.cb_url;
  const company_logo_url = bulkDataCompany.small_logo_url;
  const metadata = {
    company_name,
    company_one_liner,
    company_long_description,
    batch,
    company_location,
    company_city,
    company_city_tag,
    company_country,
    company_year_founded,
    company_website,
    company_linkedin,
    company_twitter,
    company_facebook,
    company_crunchbase,
    company_logo_url,
  };

  const chunkData: CreateChunkData = {
    chunk_html,
    group_ids: [group_id, ...groupIds],
    link,
    tag_set: tag_set.split(","),
    tracking_id,
    metadata,
  };

  await createChunk(chunkData);

  return group_id;
};

Notice that above, we keep all of the chunk's native data as metadata on the chunk resource we are creating. We do this to make our life easier when building the final frontend which users are going to interact with.

See the details of running these functions for each company's URL in the full implementation on Github at github.com/devflowinc/yc-companies/bun-scraper.

Step 4. Build a SolidJS SPA for interacting with the dataset via search

The entire frontend application for this tutorial is available at yc.trieve.ai and written in a single typescript file which you can view at github.com/devflowinc/yc-companies/blob/main/src/App.tsx.

I strongly recommend reading the SolidJS documentation to get a more in-depth understanding of how to use SolidJS in your own projects. In my opinion, it is beyond the scope of this tutorial to go into depth about how we used SolidJS to fully build out the frontend. However, if you are interested. let me know and I may write a part 2!

In this post, I primarily want to focus on how we search the dataset we indexed and stored in the previous step using Trieve.

type SearchType = "semantic" | "hybrid" | "fulltext";

const searchCompanies = async (
  curSortBy: string,
  curPage: number,
  curBatchTag: string,
) => {
  setFetching(true);
  const response = await fetch(`${apiUrl}/chunk/search`, {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      "TR-Dataset": datasetId,
      Authorization: apiKey,
    },
    body: JSON.stringify({
      page: curPage,
      query: searchQuery(),
      search_type: searchType(),
      tag_set:
        curBatchTag === "all batches" ? [] : [curBatchTag.toUpperCase()],
      highlight_results: false,
      get_collisions: false,
    }),
  });

  const data = await response.json();
  const scoreChunks = data.score_chunks;
  if (curSortBy === "recency") {
    scoreChunks.sort(
      (a: any, b: any) =>
        parseInt(b.metadata[0].metadata.batch.slice(-2)) -
        parseInt(a.metadata[0].metadata.batch.slice(-2)),
    );
  }

  if (curPage > 1) {
    setResultChunks((prevChunks) => prevChunks.concat(scoreChunks));
  } else {
    setResultChunks(scoreChunks);
  }
  setFetching(false);
};

There are 3 available search modes: semantic, hybrid, and fulltext. Semantic search uses dense vector embeddings created with a Jina model and fulltext uses sparse vectors created via SPLADEv2. Hybrid is a mix of both where the final result set is ordered with bge-reranker.

Conclusion

Build with Trieve! Join us on Matrix or Discord and tell us about what you want to build. Our community would love to hear about your ideas and plans to help you build however we can.

Finally, if you liked this tutorial, please star us at github.com/devflowinc/trieve!

Cheers! Hopefully I get to hear from you soon. Feel free to book a meeting with me here on cal.com, follow on X, or connect on LinkedIn.

Cheating at Search with LLMs

Nicholas Khami — Wed, 21 May 2025 12:55:00 GMT

Introduction

We've started doing this search technique that we've been calling "cheating at Search with LLMs" and I thought it'd be cool to talk about it. If you just want to see it live, go to demos.trieve.ai/demos/lifestraw, open devtools, navigate to the network tab, and click through the "group_oriented_search" request and subsequent tool calls.

The Problem: Search Can't Understand Intents Like Comparisons

For our Gen AI sales associate Shopify app, we wanted to make it possible to do cool things like generate a comparison table for any two products. Take this example from the brand LifeWater, which sells filterable straws. If a customer asks to "compare the Sip against the Life Straw" (two different products in their portfolio), we need to quickly look inside their catalog to determine which two products to fetch.

The challenge? No traditional keyword, semantic, or hybrid search would ever be intelligent enough without an LLM to understand the exact two products being discussed.

Our Solution: Let the LLM Do the Hard Work

So we cheat. Here's how it works:

First, we do a standard search with the user's query and get the top 20 results. Each group represents a product, and each chunk within that group is a variant of that product (like different colors or pack sizes).
Then we use a tool called "determine relevance" that asks the LLM to rank each product as high, medium, or low relevance to the query. We pass each product's JSON, HTML, description text, and title to the LLM.
The LLM examines each product and makes the call. For example, it might mark the Life Straw Sip Cotton Candy variant as "high" relevance, and the regular Life Straw as "high" relevance, while everything else gets "medium" or "low."
We then use these relevance rankings to display only the most relevant products to the user.

Making It Fast

Despite making 20+ LLM calls in the background, the experience feels instantaneous to the user thanks to semantic caching on all the tool calls. If I run the same comparison again, it's blazing fast.

Going Even Further

We extend this approach to other aspects of search:

Price Filters: We have a tool call that extracts min and max price parameters
Category Determination: For stores with predefined categories, we use LLMs to determine the right category
Format Selection: We use tool calls to decide whether to generate text or images
Context Retention: If a user follows up with "tell me more about the Life Straw's filtration," we don't need to search again - we just use the same products from before

Why This Matters

It literally feels like cheating, which is incredible. In the early days, we spent a ton of time building super high-relevance search pipelines. But with modern LLMs, that's unnecessary. You can just fetch 20 things, give the LLM the query and each fetched item, and ask it which ones are relevant.

Absolute madness. Intelligence as a commodity.

Build Search and RAG for Any Website with Firecrawl and Trieve

Daniel S. Griffin — Thu, 22 Aug 2024 19:19:00 GMT

In this guide, we will show how to use Firecrawl and Trieve to build search and RAG for SigNoz's documentation in both Python and JS.

Firecrawl's REST API can be queried to convert every page available at a URL into vector search and RAG-ready markdown.

Trieve's API can then receive chunks of the markdown docs, embed and place them into a search index, and finally be called to perform AI Search and RAG on all of the site's content.

All the code used (both node.js and Python) is also in this GitHub repo (MIT license): firecrawl-to-trieve.

Requirements - free API keys:

Firecrawl API key: Signup at firecrawl.dev
Trieve API key and dataset id: Register and setup at dashboard.trieve.ai

Setup a .env file with:

FIRECRAWL_API_KEY=
TRIEVE_DATASET_ID=
TRIEVE_API_KEY=

1. Converting Signoz docs to Markdown Chunks with Firecrawl

<TextBox> Firecrawl Docs & Github (see also their js-sdk and python-sdk) </TextBox>

We use Firecrawl to convert the pages on https://signoz.io/docs/ into markdown then save them to a json file (with a timestamp). You can adjust the crawler limit (there are only 303 pages so it falls within the free credits plan).

import FirecrawlApp from '@mendable/firecrawl-js';
import dotenv from 'dotenv';
import { fileURLToPath } from 'url';
import path from 'path';
import fs from 'fs';

const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);

dotenv.config({ path: path.resolve(__dirname, '../.env') });

// Initialize the FirecrawlApp with your API key
const app = new FirecrawlApp({ apiKey: process.env.FIRECRAWL_API_KEY });

// Define crawl parameters
const crawlUrl = 'https://signoz.io/docs/';
const params = {
    crawlerOptions: {
        limit: 500,
        maxDepth: 10,
        includes: ['docs/*'],
    },
    pageOptions: {
        onlyMainContent: true
    }
};

// Crawl the website
const crawlResult = await app.crawlUrl(
    crawlUrl,
    params,
    true, // wait_until_done
    2 // poll_interval
);

// Save the crawl result to a file (with a timestamp)
const timestamp = new Date().toISOString().replace(/:/g, '-').slice(0, -5);
fs.writeFileSync(`crawl_results_${timestamp}.json`, 
    JSON.stringify(crawlResult, null, 2));

Cleaning the Markdown

Each crawl result item has a markdown field that contains the full markdown text of the page. We did some initial data cleaning as we took our first look at the data and worked out the chunking. You can always add more cleaning later. We cut boilerplate from the end and transformed how links were presented in the metadata. This was done through tested functions that we can easily add to as we start searching.

<TextBox> Initial cleaning functions are in cleaners.py / cleaners.js, imported into the transform scripts. </TextBox>

Chunking the Markdown

For a baseline chunking approach we just want to find ways to split pages into smaller chunks, while maintaining some semantic cohesion within the chunks. Just as with cleaning, we can always refine our chunking strategy as we gain more familarity with the data and our use cases.

<TextBoxLearnMore> Learn more about chunking techniques in Greg Kamradt's "5 Levels Of Text Splitting" Notebook and Maria Khalusova's "Chunking for RAG: best practices" </TextBoxLearnMore>

Our transform script first makes chunks of any page that is less than 500 words (though this could easily be switched for a token-based limit based on the LLM of your RAG system). Then it splits longer content into chunks by top-level anchor links (e.g. [](#overview)\nOverview ) and then splitting by h3, h4, etc (e.g. ### [](#step-1-setup-otel-collector)\n). (Explicit h1-h2 (i.e. # this-is-h1 and ## this-is-h2) headings for this dataset are not indicated in the markdown. The html itself has h2 headings for both the page title and for what is rendered here as the top-level anchor link headings but it is not represented with # here.)

This transform script uses regex to recursively split longer markdown content whenever the output was greater than max_words (we used 500, determined via a crude split on whitespace) or until the split is at the max_depth (we used h4). So some chunks are longer than 500 words (25 of 972). We can later investigate lower heading-level splits, alternative split points, or manual edits for those chunks.

function processContent(pageMarkdown, pageTitle, pageLink, pageTagsSet,
                        pageDescription, maxWords = CONFIGS.maxWords, maxDepth = CONFIGS.maxDepth) {

  // Splits content into sections based on headings
  function splitContent(content, pattern) {
    const matches = content.match(new RegExp(pattern, 'g')) || [];
    return matches.map((match, i) => {
      const [, headingLink, headingText] = match.match(pattern);
      const start = content.indexOf(match);
      const end = i === matches.length - 1 ? content.length : content.indexOf(matches[i + 1]);
      return [headingLink, headingText, content.slice(start, end)];
    });
  }

  // Creates chunks from sections
  function createChunks(sections, currentTitle = '', depth = 0) {
    const localChunks = [];
    let lastChunkHeadingOnly = false;

    sections.forEach(([headingLink, headingText, sectionContent]) => {
      if (lastChunkHeadingOnly) {
        headingText = `${lastChunkHeadingOnly} - ${headingText}`;
        lastChunkHeadingOnly = false;
      }

      const chunkHtml = getChunkHtml(sectionContent, pageTitle, headingText, 0, null);

      if (chunkHtml === "HEADING_ONLY") {
        lastChunkHeadingOnly = headingText;
        return;
      }

      const fullTitle = `${currentTitle}: ${headingText}`.replace(/^:\s+/, '');
      const chunkWordCount = chunkHtml.split(/\s+/).length;
      const isWithinChunkingConstrains = chunkWordCount <= maxWords || depth >= maxDepth;
      // true if the chunk is within the word limit or we've reached max depth.
      // we don't split further at max depth, even if over word limit

      if (isWithinChunkingConstrains) {
        localChunks.push(createChunkObject(chunkHtml, pageLink, headingLink, headingText,
                                           pageTagsSet, pageTitle, pageDescription, fullTitle));
      } else {
        const subsections = splitContent(sectionContent, 
          /(\\n###+ \\[\\]\\((#.*?)\\))\\n(.*?)\\n/);
        // regex to find subsections of the current section
        if (subsections.length > 0) {
          localChunks.push(...createChunks(subsections, fullTitle, depth + 1));
        } else {
          // if no subsections, create a chunk for the current section
          localChunks.push(createChunkObject(chunkHtml, pageLink, headingLink, headingText,
                                             pageTagsSet, pageTitle, pageDescription, fullTitle));
        }
      }
    });

    return localChunks;
  }

  const topSections = splitContent(pageMarkdown, /(\\n\\[\\]\\((#.*?)\\))\\n(.*?)\\n/);
  return createChunks(topSections);
}

We also take the heading itself and add it at the top of the chunk along with the original page title.

Example: The chunk for the "Enable a Prometheus Receiver" heading on the "Send Metrics to SigNoz Cloud" page is titled: "Send Metrics to SigNoz Cloud: Enable a Prometheus Receiver"

If the split results in a chunk consists of only a heading (HEADING_ONLY), that chunk is skipped and the heading is added to the top of the subsequent chunk.

Preparing the Chunks for Trieve

The chunk will include various other fields, metadata and labels specific for Trieve:

tracking_id: parsed from the URL + the id of the heading for splits
group_tracking_ids: parsed from the URL (this will let us link the page chunks together)
tag_set: parsed from the URL segments (ex. https://signoz.io/docs/install/kubernetes/ -> ['install', 'kubernetes']; optimized for filtering)
image_urls: parsed from the markdown (the docs have both png and webp images; this will show within the Trieve search playground)
metadata: parsed from the crawl result metadata (we save the page title and description (if it exists), in case useful later; can be filtered)

function createChunk(chunkHtml, pageLink, headingLink, headingText, pageTagsSet, 
                     pageTitle, pageDescription) {
  const chunk = {
    chunk_html: chunkHtml,
    link: pageLink + headingLink,
    tags_set: pageTagsSet,
    image_urls: getImages(chunkHtml),
    tracking_id: getTrackingId(pageLink + headingLink),
    group_tracking_ids: [getTrackingId(pageLink)],
    timestamp: TIMESTAMP,
    metadata: {
      title: pageTitle + ': ' + headingText,
      page_title: pageTitle,
      page_description: pageDescription,
    }
  };

  return chunk;

<TextBox> The groups are automatically created in Trieve for each tracking ID in the group_tracking_ids array if a group with that tracking ID does not yet exist. Groups can be created (and edited) in the chunk_group route. This lets you provide a name, description, tag_set and metadata for the group. See the docs: Create or Upsert Group or Groups </TextBox>

2. Storing the Chunks in Trieve for Search and RAG

We use the Trieve api/chunk route to create the chunks in Trieve.

<TextBox> See the docs: Create or Upsert Chunk or Chunks </TextBox>

Chunks here are just parsed JSON in batches of 120.

async function loadChunks(chunks, config, upsert = false) {
  const url = `${config.basePath}/api/chunk`;
  const headers = {
    "TR-Dataset": config.datasetId,
    "Authorization": `Bearer ${config.apiKey}`,
    "Content-Type": "application/json"
  };

  chunks.forEach(chunk => chunk.upsert_by_tracking_id = upsert);

  try {
    const { data } = await axios.post(url, chunks, { headers });
    console.log(`Successfully ${upsert ? 'upserted' : 'created'} batch of ` +
      `${chunks.length} chunks to ${config.datasetId}`);
    return data;
  } catch (error) {
    console.error(`Failed to ${upsert ? 'upsert' : 'create'} batch. ` +
      `Status: ${error.response?.status}, ` +
      `Data: ${JSON.stringify(error.response?.data)}, ` +
      `Message: ${error.message}`);
    throw error;
  }
}

Testing Search and RAG quality using Trieve Playgrounds

Trying out some searches and chatting with the chunks is one way to get a look at your data and the quality of your initial cleaning and chunking steps.

After loading the chunks you can head to dashboard.trieve.ai. Here you'll find a datasets table with your ingested dataset, listing the number of chunks along with quicklinks to settings and the search, RAG, and analytics playgrounds. Both the Search and RAG Playgrounds show the chunks with their chunk_html, the links to their source pages, tracking IDs, and metadata.

Search Playground: search.trieve.ai

<TextBox> This is a search interface optimized for fully exploring the search options in Trieve. You'll see configurations for adjusting filters, choosing between search types (ex. hybrid, semantic, fulltext (SPLADE), BM25, etc.), sorting and reranking. You can also do multi-query searches and group search (returning the corresponding group for the retrieved chunks). There are additional options to set the score threshold for retrieved results, page size, stop word removal, and various settings for managing Trieve's sub-sentence highlighting. Once a search has been conducted you will also see a button to "Rate This Search" which opens a modal with a 0-10 slider and notes field. </TextBox>

Example searches to showcase the search types:

Query: visualizations (Semantic) This is a dense vector search using cosine distance vectors on Jina english embeddings), ![image](https://cdn.trieve.ai/blog/firecrawl-and-trieve/visualizations_query_signoz_semantic.png)

Query: visualizations (Fulltext) Notice how the retrieval model for our fulltext search type, SPLADE, is returning chunks containing words closer to the query rather than the broader semantics. ![image](https://cdn.trieve.ai/blog/firecrawl-and-trieve/visualizations_query_signoz_fulltext.png)

Query: visualizations (Hybrid) This takes in results from both fulltext and semantic and re-ranks with a cross-encoder model (here the BAAI/bge-reranker-large). ![image](https://cdn.trieve.ai/blog/firecrawl-and-trieve/visualizations_query_signoz_hybrid.png)

Group Search: Internally this uses the chunk_group/group_oriented_search route.

Query: OpenTelemetry FastAPI example (Hybrid) ![image](https://cdn.trieve.ai/blog/firecrawl-and-trieve/fastapi_example_query_signoz_semantic_group.png)

RAG Playground: chat.trieve.ai

This is a question-and-answer chat interface to showcase and explore RAG in your dataset with Trieve. It has everything you are looking for: input at bottom, streaming response from the LLM, and documents (the retrieved chunks) used by the LLM in its response on the right-hand side. You can regenerate a response, ask follow-on questions, start new conversations, and open any of the retrieved chunks. The various RAG-relevant parameters can be adjusted in the dataset settings in the dashboard: LLM, HyDE prompt (optional), System Prompt, RAG prompt, and more. We support models available on OpenRouter: openrouter.ai/docs/models

What is HyDE?

Hypothetical Document Embeddings (HyDE) is a retrieval method where an LLM first generates a hypothetical document that might address a user's query, and then that document is used in a semantic search (instead of the original query) to retrieve documents from the dataset. Read more in Luyu Gao et al.'s "Precise Zero-Shot Dense Retrieval without Relevance Labels".

Example interactions on the RAG interface:

Question: What is SigNoz?

![image](https://cdn.trieve.ai/blog/firecrawl-and-trieve/what_signoz_chat_signoz.png)

Question: What are the benefits of OpenTelemetry? ![image](https://cdn.trieve.ai/blog/firecrawl-and-trieve/open_telemetry_benefits_chat_signoz.png)

Conclusion

Firecrawl and Trieve are a powerful combination for quickly building search and RAG systems. Get started today!

Firecrawl: firecrawl.dev
Trieve: trieve.ai
Github repo: firecrawl-to-trieve

History of HackerNews Search: From 2007 to 2024

Nick Khami — Mon, 12 Aug 2024 19:17:00 GMT

We at Trieve are going to be launching a search engine for HackerNews with some additional features soon and thought it would be worth studying the history of HN search before finalizing things. Here's what we found!

Update!

We launched at the end of August! Check it out at hn.trieve.ai.

Here's a writeup on the launch (HN post).

Note: Did the research using our own HN search engine! :)

Note: I was happy looking at these old HN posts and seeing that so many of the comment/post'ers from the early days of HN were, or eventually became, founders of YC companies.

First Gen | 2007-2011

1. Search News YC (bigheadlabs) on March 17, 2007

Written and shared by Jason Yan, Founder/CTO of Disqus (S07) (aka jsonyan), on March 17, 2007. Can still be viewed here on the internet archive.

I assume there was some indexing logic being done on the DJango server that Jason used.

2. ycsearch.com on June 27, 2007

Quickly hacked together by Keven Lin (YC S07) (aka keven) on June 27, 2007.

I could not find a screenshot on Internet Archive, but Keven explained he built it with cse.google.com.

3. Search 'Hacker News' (trk7.com) on September 18, 2007

Created and shared by Kesevan (aka cosmok) on September 18, 2007.

Can see the UI at this link on the internet archive.

In the text of the original post (see here), cosmok explains that he built it using Yahoo's search API.

4. searchyc.com (independent) on Dec 31, 2007 | First with Some Staying Power

Independently created and shared by Mike Cheng (aka chengmi) and Alaska Miller (aka alaskamiller) on Dec 31, 2007.

I surmise from the comment on the post that the motivation here was ycsearch being limited in terms of HN-specific filters and the bigheadlabs one being un-maintained.

Judging from the last paulg comment on this thread it seems like, similar to ycsearch, it was built using cse.google.com.

The HN community seemed to get a lot of value out of it as in a HN thread posted when it went down on June 1 of 2011 there are multiple users explaining how important it was to them:

iheartmemcache: This service is a major component of this community; as such, I'll host this on whatever metal you need. My contact information is in my profile. Ping me on G-talk and we can have this sorted out by the morning (if you're in PST).

bkrausz: What kind of traffic does SearchYC get? Is a $40/mo Linode not sufficient? I would gladly pay that (or be content with some Google ads in the right bar). Hell, I'd even maintain the site...it's a great service.

g123g: Hopfully you will be able to bring it back soon. SearchYC.com is the best way to search the treasure trove that HN has become.

Worth mentioning that HNSearch (mentioned further below) was up by this point in time. Judging by comments on the shutdown post it seems like traffic was somewhat split:

swombat: What's wrong with http://www.hnsearch.com ?

evangineer: Just got zero hits on a search that I know there is at least one result for. Same search worked fine on searchyc.com a few days ago.

Second Gen - Octopart/ThriftDB-powered HNSearch | 2011-2014

The official launch of HNSearch was posted by Paul Graham, founder of Y-Combinator itself (aka pg), and Andres Morey, founder of Octopart (W07) (aka andres), separately on June 4, 2011. Andres posted it as an API contest here to build the best thing on top of the HNSearch API where the winner would get a 27-inch Dell monitor. PG posted it as an official announcement on HN here which linked to ycombinator.com in a now only archive-available page.

Building search for HN has certainly been a trial for us and we felt validated seeing that PG first mentioned the Octopart guys using ThriftDB to make this in 2007 4yrs before it released.

I think the best part of HNSearch was that third-party applications were built on top of it. It seems, judging by the HNSearch shutdown post, that it was well-loved by HN users and also well-replaced by Algolia.

clamprecht: Can someone outline the benefits of the new one over the old one? When I first tried the new one, the UI was severely lacking. I saw the fixed a few things, but I haven't evaluated it again. I don't always use the HN search engine, but when I do, it's usually very helpful. I'd hate to lose that.

swahI noticed the new one is much faster..

Third Gen - Algolia powered search | 2014-current

The first HN post I was able to find mentioning Algolia HN search was Ask HN: What do you think about our last HN Search update? on Jan 26, 2014 posted by Julian Lemoine, founder/CTO of Algolia (W14) aka jlemoine.

Algolia's ability to get HN search up so quickly is really impressive. If you look at the Github repo it seems like they started in Sep 2013 and released in Jan 2014.

We also took about 6 months to get everything up having started in Feb 2024 and releasing in Aug 2024. I can say now, with firsthand experience, that timeline is not easy to operate on. Especially given certain devtooling was less mature in 2013.

Algolia asked the community for feedback post-launch in 2014 and implemented several improvements including additional filter types and improved indexing speed in late 2023. We think it's incredibly accurate for keyword search and has all the filters and options that we would want.

Honorable Mentions during the Algolia era

hackersearch.net by jnnnthnn posted May 2024 | semantic search engine using OpenAI embeddings
deephn.org by wolfgarbe posted April 13, 2021 | full-text search of both HN posts and linked webpages
hackernews.demo.vectara.com by ofermend posted July 2024 | semantic search for past 6mths of data
searchhacker.news by isoprophlex posted April 2024 | keyword search over discussions re-ranked by dense semantic vectors
hn.lixiasearch.com by larose posted February 2024 | unknown data level and indexing strategy
orangewords.com by cmcollier (not posted to HN yet) | all of HN indexed in Vespa with RAG

How to Build and Use Agentic RAG

Nick Khami — Fri, 30 May 2025 16:39:00 GMT

The Problem: RAG That Searches for Everything

Traditional RAG systems have a glaring issue: they search for everything. User asks about pizza recipes? Search the knowledge base. Wants to know the weather? Search again. Having a casual conversation? Yep, another search.

This shotgun approach leads to:

Irrelevant context cluttering the LLM's input
Higher costs from unnecessary searches and longer prompts
Slower responses due to constant database hits
Confused AI trying to make sense of unrelated documents

What if the AI could decide for itself when it actually needs to search?

Enter Agentic RAG: The LLM Calls the Shots

Agentic RAG flips the script. Instead of automatically searching on every query, we give the language model tools it can choose to use. Think of it like handing someone a toolbox—they'll grab a hammer when they need to drive a nail, not when they're stirring soup.

In our implementation, the LLM gets two main tools:

Search tool: "I need information about X"
Chunk selection tool: "I'll use these specific documents"

The magic happens when the AI decides it needs more information. Only then does it reach for the search tool. If you are only interested in the source code (Rust btw 😉), check it out on Github here.

Try It Yourself: Two API Calls to Agentic RAG

Want to add intelligent RAG to your application right now? You can use our agentic search system with just two simple API calls. No need to build anything from scratch.

Step 1: Create a Chat Topic

First, create a topic to hold your conversation. Refernece the full documentation at docs.trieve.ai/api-reference/topic/create-topic for more details.

curl -X POST "https://api.trieve.ai/api/topic" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "TR-Dataset: YOUR_DATASET_ID" \
  -H "Content-Type: application/json" \
  -d '{
    "owner_id": "user_123",
    "name": "My Chat Session"
  }'

This returns a topic with an id that you'll use for the conversation.

Step 2: Send Messages with Agentic Search

Now send your message and let the AI decide when to search. To see the full documentation, visit docs.trieve.ai/api-reference/message/create-message.

curl -X POST "https://api.trieve.ai/api/message" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "TR-Dataset: YOUR_DATASET_ID" \
  -H "Content-Type: application/json" \
  -d '{
    "topic_id": "TOPIC_ID_FROM_STEP_1",
    "new_message_content": "How do I configure authentication?",
    "use_agentic_search": true
  }'

That's it! The system will:

Analyze if your question needs a knowledge base search
Search intelligently only when needed
Stream back responses in real-time
Show you exactly which documents were used

The magic is in that use_agentic_search: true parameter—it activates the intelligent search behavior described below.

How We Built It: One Smart API Route

Our entire agentic RAG system lives in a single function: stream_response_with_agentic_search. It's surprisingly straightforward once you break it down.

Step 1: Setting Up the Conversation

// Simplified: Prepare messages for the LLM
let mut openai_messages: Vec<ChatMessage> = messages
    .iter()
    .map(|message| ChatMessage::from(message.clone()))
    .collect();

// Add the current user query
openai_messages.push(ChatMessage::User {
    content: ChatMessageContent::Text(user_message_query.clone()),
    name: None,
});

Nothing fancy here—we take the conversation history and current user message, then format them for the LLM. If the user included images, those get added too.

Step 2: Creating the Toolbox

This is where it gets interesting. We define tools that the LLM can call:

// Simplified: Define tools the LLM can use
let tools = vec![
    ChatCompletionTool {
        function: ChatCompletionFunction {
            name: "search".to_string(),
            description: "Search for relevant information in the knowledge base",
            parameters: json!({
                "type": "object",
                "properties": {
                    "query": {
                        "type": "string",
                        "description": "The search query to find relevant information"
                    }
                }
            }),
        },
    },
    ChatCompletionTool {
        function: ChatCompletionFunction {
            name: "chunks_used".to_string(),
            description: "Tell the user which chunks you plan to use",
            parameters: json!({
                "type": "object",
                "properties": {
                    "chunks": {
                        "type": "array",
                        "items": {"type": "string"}
                    }
                }
            }),
        },
    }
];

The search tool lets the AI query our knowledge base when it needs information. The chunks_used tool allows the AI to explicitly state which retrieved documents it's actually using—no more mystery about where answers come from.

Step 3: The Conversation Loop

Here's where the agent behavior emerges:

// Simplified: Main conversation loop
loop {
    // Get response from LLM
    let response = client.chat().create(parameters.clone()).await?;

    // Add assistant message to conversation
    conversation_messages.push(response.message.clone());

    // Check if AI wants to use tools
    if let Some(tool_calls) = response.tool_calls {
        for tool_call in tool_calls {
            match tool_call.function.name.as_str() {
                "search" => {
                    // AI decided it needs to search
                    let (results, formatted_results) = handle_search_tool_call(
                        tool_call,
                        dataset,
                        pool,
                        redis_pool,
                        dataset_config,
                        event_queue,
                    ).await?;

                    // Add search results back to conversation
                    conversation_messages.push(ChatMessage::Tool {
                        content: formatted_results,
                        tool_call_id: tool_call.id,
                    });

                    searched_chunks.extend(results);
                }
                "chunks_used" => {
                    // AI specified which chunks it's using
                    let chunks_to_use: Vec<String> = parse_chunks_used(&tool_call)?;

                    // Filter to only keep specified chunks
                    searched_chunks.retain(|chunk| {
                        chunks_to_use.contains(&chunk.id.to_string())
                    });
                }
                _ => {}
            }
        }
        // Continue conversation with tool results
        continue;
    } else {
        // No tool calls - we have the final response
        break;
    }
}

This loop is the heart of the agentic behavior. The LLM generates a response, and if it includes tool calls, we execute them and add the results back to the conversation. The loop continues until the LLM provides a final answer without requesting any tools.

Step 4: Real-Time Streaming

For the best user experience, we stream responses in real-time. This gets trickier because tool calls can interrupt the stream:

// Simplified: Streaming with tool support
while let Some(response_chunk) = stream.next().await {
    match response_chunk {
        Ok(chunk) => {
            // Stream content to user
            if let Some(content) = chunk.content {
                tx.send(content).await;
            }

            // Handle tool calls mid-stream
            if chunk.finish_reason == Some("tool_calls") {
                // AI wants to use tools - pause streaming
                tx.send("[Searching...]").await;

                // Execute tool calls
                handle_tool_calls(&chunk.tool_calls).await;

                // Resume streaming with updated context
                continue;
            }
        }
        Err(e) => break,
    }
}

The user sees responses appear in real-time, with helpful indicators like "[Searching...]" when the AI decides it needs more information.

What Makes This Actually Work

Tool Choice Intelligence

The LLM learns when to search based on the conversation context. If someone asks "What's the weather like?", it typically won't search a knowledge base about product documentation. But if they ask "How do I configure authentication in your API?", it will search for relevant docs.

Transparency Through Chunk Selection

The chunks_used tool solves a major RAG problem: knowing where answers come from. Instead of the system dumping 20 documents into context and hoping for the best, the AI explicitly states which documents it's using.

Streaming with Interruptions

Most agentic systems batch everything—ask, search, think, respond. Our streaming approach provides immediate feedback while still allowing the AI to search mid-conversation.

Real-World Performance

Since implementing this approach, we've seen:

60% reduction in unnecessary searches
40% faster responses on queries that don't need retrieval
Higher accuracy because context is more relevant when searches do happen
Better user experience with transparent document usage

The system handles everything from simple greetings (no search needed) to complex technical questions (multiple targeted searches) seamlessly.

The Trade-offs We Made

Like any system, this approach has trade-offs:

Pros:

Intelligent search decisions
Transparent source attribution
Real-time streaming responses
Lower costs and latency

Cons:

Slightly more complex than traditional RAG
Requires tool-capable LLMs
Multiple round trips for complex queries

What's Next

We're continuously improving the system:

Smarter tool descriptions that help the LLM make better decisions
Multi-step reasoning for complex queries requiring multiple searches
Domain-specific tools beyond just search
Better chunk selection with reasoning about why specific documents are relevant

The Bottom Line

Agentic RAG isn't about adding complexity—it's about adding intelligence. By letting the LLM decide when and what to search for, we've built a system that's both more efficient and more effective than traditional approaches.

The beauty is in the simplicity. One API route, a few well-defined tools, and suddenly your RAG system becomes an intelligent agent that knows when to look things up and when to just have a conversation.

Want to see it in action? Check out our implementation at github.com/devflowinc/trieve or try it yourself at docs.trieve.ai/api-reference/message/create-message.

Building something similar? We'd love to hear about your approach and any challenges you've faced. The future of RAG is agentic, and we're excited to see what the community builds.

The Future of Work: Commanding Armies of Parallel LLMs

Thu, 22 May 2025 17:25:00 GMT

import ParallelClothesLabeling from "@components/ParallelClothesLabeling";

Massive parallel LLM inference is underrated.

If you use AI today, you're likely:

Asking ChatGPT or Claude to do something for you
Navigating away to thumb on another task for just long enough
Navigating back to judge its output to then either:
a. Accepting it, implementing the solution, and moving on
b. Rejecting it and trying to reason with the AI again, and again, and again...

This is a waste of time! The future of work looks like StarCraft or Age of Empires.

Imagine you have an allocation of 10,000 requests—rather than a single prompt—at 100,000 tokens per second. (For context, the average GPT interaction uses 50-150 tokens per turn.) You're directing this computational firepower to solve problems and create solutions, acting as the commander with agentic systems as your units. If the point of battle is to win, you need asymmetric advantages. But this isn't just about sheer firepower or brute force—though that's part of it. The real power lies in your ability to direct and manage these forces strategically to maximize their effectiveness.

Like any real working environment, you don't spread capacity evenly across 200 different tasks. You provision agents to match the problem's nature. Some tasks stand alone, while others form clusters—interconnected webs where complexity and complication (two distinct beasts) hide cloaked in an ether of ethers.

Yes, each LLM has some probability of finding a solution. But you don't just want one correct solution per problem. You want enough to get multiple, then mix and match them into one extremely high quality solve. Think of it like Monte Carlo sampling, you're hedging against randomness as you explore the solution space.

Data labeling is a fantastic base case example

As a familiar and persistent example, we built a demo app that categorizes images of clothing using parallel LLM calls. As a search company, being able to structure arbitrary datasets across different sources greatly improves our ability to index and manipulate information into awesome AI experiences for our customers' customers.

But here's what's actually cool: you could photograph all your belongings and have AI instantly sort which items are worth more than $50. With humans, you need dozens of people working in parallel to finish in reasonable time.

Google spent years tricking hundreds of millions of people into labeling data through CAPTCHAs. Now you can leverage the years of human effort that went into this and accomplish similar tasks for much less in a few hours. As you might also assume, this will get faster and cheaper as LLMs improve.

Parallel compute should be all of the time

We should be using parallel compute for everything AI. We should be supervisors, not spectators.

Right now, as a software engineer, I open my IDE, prompt an agent to edit code, then watch it work. Strong WALL-E fitless human vibes; disengaged, passive.

I don't want this. I want to be in flow state, hyper-engaged and forward-leaning. I want to feel and produce like Ender commanding an entire fleet.

It should be trivial to provision multiple LLMs with high variance to tackle tasks in parallel. If AI can generate product photos, I want to pick 3 styles and deploy 6 agents, 2 per style-generating image sets—simultaneously. If something's going wrong, I can zoom in and intervene while they work. The feedback I give one agent can be immediately applied to the others, allowing them to adjust their outputs in real-time.

This is similar to a strategy some employ on Fiverr when they hire multiple freelancers for the same project to judge the quality of their work and pick the best candidate.

New tools are needed

You can sort of replicate this with git worktrees for programming, but it's clunky. We need new software designed for managing parallelized general intelligence.

What we have now feels like a horseless carriage, we're mimicking old patterns instead of embracing new possibilities. The future UX for AI isn't watching a single agent work. It's commanding hundreds simultaneously.

Maybe these tools will be vertical-specific. Maybe they'll be general-purpose. Either way, I'm excited to see what we build and get it in your hands.

Streaming LLM assistant completions with the OpenAI API and Rust Actix-Web

Denzell Ford — Thu, 08 Aug 2024 18:54:00 GMT

We were tired of using Javascript for our backend services when we started building Trieve. We wanted something better, something faster, something safer, something rusty. Our main motivation behind choosing to use rust was for the learning experience behind it.

Why Actix

When looking at what framework to use we had two choices actix_web or rocket. We chose to use actix and actix_web because we heard that it was faster than rocket from benchmarks.

Our Naive Solution

Streaming data with Actix

The first thing we saw on how to stream data in actix_web was using tokio's built-in mpsc_channels.

pub type StreamItem = Result<Bytes, actix_web::Error>;

pub async fn stream_response(messages: Vec<Message>, pool: web::Data<Pool>)  {
    let (tx, rx) = mpsc::channel::<StreamItem>(1000);
    let receiver_stream: ReceiverStream<StreamItem> = ReceiverStream::new(rx);
    ...
    // Send data at some point
    let _ = tx.send(Ok("data".into())
    ...

    // Return Result<HttpResponse, actix_web::Error> to client
    Ok(HttpResponse::Ok().streaming(receiver_stream))
}

Getting Completions from openAI

For streaming we used the openai_dive crate. Ensure you enable the stream feature flag.

According to their documentation, you can stream data like this:

...
    let client = Client::new("sk-xxxxxxxxxxxxxxxx");

    let parameters = ChatCompletionParameters {
        messages: vec![
            ChatMessage {
                role: Role::User,
                content: "Hello!".to_string(),
                name: None,
            },
            ...
        ],
        ...
    };
    let mut stream = client.chat().create_stream(parameters).await.unwrap();
    while let Some(response) = stream.next().await {
        match response {
            Ok(chat_response) => chat_response.choices.iter().for_each(|choice| {
                if let Some(content) = choice.delta.content.as_ref() {
                    print!("{}", content);
                }
            }),
            Err(e) => eprintln!("{}", e),
        }
    }
...

Bringing both together

Streaming data won't get saved to open AI, so as we iterate through the messages we should keep track of the full message so we can store it to the database when streaming finishes. Giving us this as our final function

pub async fn stream_response(
    messages: Vec<Message>,
    pool: web::Data<Pool>
) -> Result<HttpResponse, actix_web::Error> {

    let (tx, rx) = mpsc::channel::<StreamItem>(1000);
    let receiver_stream: ReceiverStream<StreamItm> = ReceiverStream::new(rx);

    let open_ai_messages: Vec<ChatMessage> = messages
        .iter()
        .map(|message| ChatMessage::from(message.clone()))
        .collect();
    let client = Client::new("sk-xxxxxxxxxxxxxxxx");

    let parameters = ChatCompletionParameters {
        messages: open_ai_messages,
        ...
    };

    let mut response_content = String::new();
    let mut completion_tokens = 0;
    let mut stream = client.chat().create_stream(parameters).await.unwrap();

    while let Some(response) = stream.next().await {
        match response {
            Ok(chat_response) => {
                completion_tokens += 1;

                log::info!("Got chat completion: {:?}", chat_response);

                let chat_content = chat_response.choices[0].delta.content.clone();
                if chat_content.is_none() {
                    log::error!("Chat content is none");
                    continue;
                }
                let chat_content = chat_content.unwrap();

                let multi_use_chat_content = chat_content.clone();
                let _ = tx.send(Ok(chat_content.into())).await;
                response_content.push_str(multi_use_chat_content.clone().as_str());
            }
            Err(e) => log::error!("Error getting chat completion: {}", e),
        }
    }

    let completion_message = Message::from_details(
        response_content,
        ...
    );

	// Lets hope this finishes in the background
    let _ = web::block(move || {
	create_message_query(completion_message, &pool)
    }).await?;

    Ok(HttpResponse::Ok().streaming(receiver_stream))
}

This has issues though, when we ran this, it returned to the client once the streaming context completed. Not in a streaming context.

The big issue that we didn't see here is that calling await on the stream at all will block the main thread from returning until it has fully resolved.

This lead us down a rabbit hole of trying to put this while loop in a different process. This proved difficult because we wanted to spawn an asynchronous function on a separate thread, while Actix is much better suited for spawning synchronous functions on separate threads.

This had other larger issues because mpsc::channel channels don't implement the Send trait that is vital for it to be sent across threads and frankly we didn't have the knowledge to implement the Send trait ourselves

What went wrong

Eventually we went back a few steps and looked at it a different way. Instead of looking how to stream to the client first, then how to put the OpenAI messages into that stream. We looked at how to push the OpenAI stream to the client. Then how to push that data onto the server. Worse case-scenario, we have the client send back the completed message to the server.

Looking at tokio_stream I found the StreamExt trait which allows you to map a stream as if it was an iterator. This ended up giving us a much more concise function that looked very aesthetic (if you ask me).

How to properly stream

pub async fn stream_response(
    messages: Vec<models::Message>,
    pool: web::Data<Pool>
) -> Result<HttpResponse, actix_web::Error> {

    let open_ai_messages: Vec<ChatMessage> = messages
        .iter()
        .map(|message| ChatMessage::from(message.clone()))
        .collect();

    let client = Client::new("sk-xxxxxxxxxxxxxxxx");

    let parameters = ChatCompletionParameters {
        messages: open_ai_messages,
	    ...
    };

    let stream = client.chat().create_stream(parameters).await.unwrap();

    Ok(HttpResponse::Ok().streaming(
        stream.map(|response| -> Result<Bytes, actix_web::Error> {
            if let Ok(response) = response {
                let chat_content = response.choices[0].delta.content.clone();
                return Ok(Bytes::from(chat_content.unwrap_or("".to_string())));
            }
            log::error!("Something bad happened");
            Err(ServiceError::InternalServerError.into())
        })
    ))
}

Great, now its streaming, but we removed core functionality that pushed the completed message to the database. We need to both process data while sending it to the client. So... we went back to channels.

Actix Arbiter

In order for this to truly work how we want, we need to bring in another actix primitive, `Arbiters. Arbiters spawn a process on a different thread, on this separate thread we will need to get the completion messages via a channel and write to the database once the full response is received.

Arbiter::new().spawn(move {
    // Do stuff on other thread
});

mpsc_channels can't be sent across threads, we instead used a different crate crossbeam-channel. This has channels that can be sent across threads.

This time when we looked at channels, we had the main thread transmit data over to the child thread. This was easier to do while mapping the stream before data gets sent down to the client.

Proper solution

pub async fn stream_response(
    messages: Vec<models::Message>,
    pool: web::Data<Pool>,
) -> Result<HttpResponse, actix_web::Error> {

    let open_ai_messages: Vec<ChatMessage> = messages
        .iter()
        .map(|message| ChatMessage::from(message.clone()))
        .collect();

    let open_ai_api_key = std::env::var("OPENAI_API_KEY")
							.expect("OPENAI_API_KEY must be set");

    let client = Client::new(open_ai_api_key);

    let parameters = ChatCompletionParameters {
        messages: open_ai_messages,
	    ...
    };

    let (sending_chan, receiving_chan) = crossbeam_channel::unbounded::<String>();
    let stream = client.chat().create_stream(parameters).await.unwrap();

    Arbiter::new().spawn(async move {
        let chunk_v: Vec<String> = receiving_chan.iter().collect();
        let completion = chunk_v.join("");

        let new_message = models::Message::from_details(
            completion,
            ...
        );

		// Write to database since we are in an arbiter no need to block
        let _ = create_message_query(new_message, &pool);
    });

    Ok(HttpResponse::Ok().streaming(stream.map(
        move |response| -> Result<Bytes, actix_web::Error> {
            if let Ok(response) = response {
                let chat_content = response.choices[0].delta.content.clone();
                if let Some(message) = chat_content.clone() {
				    // Send data to arbiter
                    sending_chan.send(message).unwrap();
                }
                return Ok(Bytes::from(chat_content.unwrap_or("".to_string())));
            }
            Err(ServiceError::InternalServerError.into())
        },
    )))
}

Conclusion

There are 2 major learnings we got from this experience.

The first is that in Rust its best to think of a solution that uses functional programming first. Then move on to a more imperative solution.

The second is that Rust still has very few resources on what is best to use. We had the idea to use crossbeam_channel from a random forum comment here that was 2 years old. We were very skeptical of even using it because it felt like a random third party hack that might not have worked with tokio's runtime. We only chose it out of desperation. That's the main motivator behind this blog post.

The current code has a few edits to it, but is essentially the same, you can navigate to it on the link below.

Good Luck Rustaceans and Happy Hacking!

GITHUB: https://github.com/devflowinc/trieve

Specific File: Here

PGVector's Missing Features

Introducing Trieve Vector Inference

Trieve Vector Inference (TVI), our solution for fast, unmetered embedding vector inference in your own cloud or on your own hardware, is now generally available as a standalone product!

Building AI features at scale exposes two critical limitations of cloud embedding APIs: high latency and rate limits. Modern AI applications require better infrastructure.

The platform supports any embedding model, whether it’s your own custom model, a private model, or popular open-source options. You get the flexibility to choose the right model for your use case while maintaining complete control over your infrastructure.

We put together TVI to eliminate these bottlenecks for our own core product. It’s served billions of queries across billions of documents. After requests from others, we’ve sanded it down, wrote up some docs, and are now making it available for all. You can even get it on AWS Marketplace!

So, just how good is TVI?

To start, here are our benchmarks:

At 1000 RPS, your P99 latency with cloud-based embeddings will range between 23.59 to 27.03 seconds. With Trieve, we’re still measuring in milliseconds. It’s literally an order of magnitude faster.

Additionally, check out the failed requests. Notice that we ran 30,000 with no failures. Without TVI, you’re only making ~7,000. If you’re going through Sagemaker, it’s around ~3,000.

TVI is a simple solution that solves two problems

Rate limits

Rate limits force you to implement complex batching and queueing. Now you’re allocating development time on workarounds instead of building core product. This crucial part of your pipeline should not feel like an exercise. We’ve seen teams build elaborate workarounds:

Multiple API keys rotated on a schedule
Distributed rate limit tracking across microservices
Complex retry logic with exponential backoff
Request queuing systems that rival Kafka in complexity

One customer had an entire Kubernetes cluster dedicated to managing their embedding pipeline - not because of compute needs, but just to handle rate limit coordination across their services. Another built a “request budget” system that required teams to reserve embedding capacity days in advance.

None of this complexity adds value to your product. It’s pure overhead, stealing engineering time from features that actually matter to your users.

High Latency

High latency robs your magic. When every embedding request takes 300ms+ round trip, your real-time features aren’t really real-time anymore.

The cascading effects are brutal:

Search results that lag behind user typing
Recommendations that feel disconnected from user actions
Chat experiences with noticeable “thinking” delays
Batch processes that take hours instead of minutes

Add in retries for rate limits and your “real-time” feature is now consistently 1+ second behind.

We hope you like the TVI DX

Most AI dev tools feel like they are at home in Jupyter notebooks or made for quick prototyping. The platforms hyperscalers and others built to onboard the masses onto AI were not designed to handle large scale.

Most production-grade software seems to fall into two buckets. It’s either 1) expensive and requires intense engagement with a vendor, and is limited in scope or 2) painful to self-host, let alone begin to productionize. We’ve talked to some folk who spent weeks building and tweaking their own servers. Others who went the enterprise route and now have to maintain annoying shadow infrastructure. Pain!

Trieve is a dev-first company. This means all engineers. DevOps, frontend, backend, everyone. We really think TVI is one of those rare solutions that works for tinkerers and giants alike. General support is available over 12 hours per day over email, Discord, Slack, and our office line.

Getting Started

You have two options for getting started with TVI. You can 1) buy a license from us and self-host it or 2) deploy it via AWS through AWS Marketplace. Self-hosting gives you maximum control, while AWS Marketplace provides the fastest path to production and still a ton of control.

Our deployment process is streamlined for both paths:

AWS: Deployment via Helm through Marketplace
Self-hosted: Docker images and clear documentation for any environment

We back TVI with a 15-day integration guarantee - if you can’t get it running in your environment, we’ll refund your fees. We’ve had teams go from zero to production in under an hour.

The Future of Vector Inference

We’re committed to making vector inference more accessible, faster, and more cost-effective for teams building AI features. Our roadmap includes:

Enhanced monitoring and observability (custom Grafana dashboards coming soon)
Support for more cloud providers (GCP and Azure coming soon)
Additional model optimizations (including quantization and pruning)
Advanced scaling features (automatic horizontal scaling based on load)

Plus, we’re working on some exciting features we can’t talk about yet. If you think today’s latency numbers are good, stay tuned!

Success Story: BillTrack50

Federico Chávez Torres — Tue, 03 Sep 2024 19:22:00 GMT

About BillTrack50

BillTrack50 is the search engine for democracy. The platform is the OS for the nation’s non-profits, advocacy groups, and corporate government affairs departments. A competitive space, the BillTrack50 development team works hard to unlock advantages and leverage for their customers, at scale.

BillTrack50’s values are evident in their free Citizen tier, where anyone can empower themselves with historical and current legislation to direct and inform their action in our democracy. After reaching out and connecting on a desire to use AI to make bill discovery and tracking 10x easier, they engaged with Trieve to make it happen.

AI Goals

![Similar Bills in Action](https://cdn.trieve.ai/BT50%20AI_720.gif)

One of the core experiences to the BillTrack50 platform is to discover and track bills relevant to their interests. With tens of thousands of new bills to discover every legislative cycle, organizations were spendings countless hours per year manually querying for bills related to theirs. Not to mention the anxiety of being uncertain one has discovered all of the relevant bills and that one has not slipped past their filters and keywords.

Simply put, there are ways to write legislation about 'gun control' or 'abortion' without the bill actually containing these words. Vectors can be very powerful here, when done correctly. The goal was to create an 'easy button' for users to find bills similar to a given piece of legislation with a single click.

With most of the support being done over email, it took the team one sprint to build, test, evaluate the Trieve-based solution against others, and deploy it to a smaller group of customers. After a successful trial period, it was immediately well-received by the greater userbase.\

Trieve has been more than a vendor to us; they've been a true partner in helping us realize our AI aspirations. From the outset, their team demonstrated exceptional patience and expertise, guiding us through every step of the process. Their timely support and advice, rooted in deep experience and knowledge, was crucial to our success. We were actually able to roll out our new AI-driven features well ahead of schedule. For any company looking to add AI features into their product, Trieve is the go-to expert you can trust.
\

Karen Suhaka - Chief Catalyst

Results

We are grateful to the BillTrack50 team and their users for informing critical development and hitting our service with frequency. Trieve is proud to work with such a cornerstone company and excited to see what they build next.

Search, recommendations, and RAG share fundamental mechanics. Trieve is the API and the team to get your features built, shipped, and deployed at scale for intense usage. If this is you, book a call here.

Success Story: Mintlify'

Federico Chávez Torres — Thu, 05 Sep 2024 19:25:00 GMT

About Mintlify

Mintlify is the modern standard for documentation. With its developer-forward product, they make it easy to build beautiful documentation whether you're a large enterprise or just getting started.

Trieve itself is a proud Mintlify customer: checkout the docs here.

After reaching out and discussing their technical requirements and desired outcomes for search and RAG, we engaged to support the team's build.

Building, Evaluating, and Implementing

We first built a proof-of-concept for the Mintlify team to evaluate the out-of-the-box search quality of Trieve. After drawing out an implementation and proof schedule, it took the team one sprint to build, test, evaluate the Trieve-based solution against others, and deploy it to a smaller group of customers.

After a successful testing period, large-scale search improvements launched as part of Mintlify's Summer launch week.

Because Trieve is an all-in-one solution, control over chunking, highlighting, and analytics give the team extra levers and knobs to ensure customer satisfaction. Since then, Trieve-powered RAG has been rolled out as well.

Acknowledgements

We are grateful to the Mintlify team for informing critical development and hitting our service with intense frequency.

Search, recommendations, and RAG share fundamental mechanics. Trieve is the API and the team to get your features built, shipped, and deployed at scale. If this is you, book a call here.

Trieve Secures $3.5M to Power AI Search and RAG

Nick Khami — Wed, 11 Sep 2024 12:34:00 GMT

SAN FRANCISCO, CA (September 11, 2024) - We at Trieve are thrilled to announce our oversubscribed $3.5M round led by Root Ventures! Our backers include Y Combinator, Soma Capital, Kulveer Taggar, Transpose Platform, and a distinguished group of strategic angel investors such as JJ Fliegelman, Richard Aberman, Rajiv Ayyangar, Jenny Fleiss, and Rohan Das.

Myself and Denzell Ford founded Trieve because we wanted infrastructure purpose-built for AI. Trieve makes it easy for developers building AI applications to determine which retrieval techniques best fit their problem and productionize with confidence.

We are looking to power every industry building with AI. New generations of discovery technology are revolutionizing user experiences and we are on the frontier.

This fresh injection of capital will be directed towards bolstering our sales strategy and strengthening customer success outcomes across e-commerce, ERP, social media platforms, and beyond. With current customers like Mintlify, BillTrack50, and AmLaw100 Firms, over 16,000 search bars are powered by Trieve.

Lee Edwards, our lead partner from Root Ventures, and us share a frustration and desire to enable quality retrieval across the internet: "Everyone thinks in-app search sucks, and that's because we are used to the intelligent, AI-based search that has improved over time from search engines like Google. But inside apps, even the large incumbents aren't doing much more than text matching. RAG and other AI-based techniques deliver the kind of user experience that can actually help products see lift from their search features, and Trieve makes these advanced techniques easy for any engineering team to implement."

I am grateful to our partners, team, community, and customers. We will continue to build the best-possible retrieval infrastructure with hacker spirit! Star us at github.com/devflowinc/trieve!

About Trieve:

Incubated in 2023 through the FUTO Fellows Program and accelerated by Y Combinator (W24), Trieve is powering search, recommendations, and AI-driven features for businesses across diverse sectors, including e-commerce, ERP, and social media platforms. If you want to chat about building AI features into your product, book a meeting with us here.

For media inquiries, please contact: media@trieve.ai.

We are thrilled to announce that trieve is being acquired by Mintlify! 🎉

Nicholas Khami — Wed, 16 Jul 2025 21:28:00 GMT

We are thrilled to announce that Trieve is being acquired by Mintlify! 🎉

We started Trieve with a mission: to make it easy for developers to build AI applications that can retrieve and reason over large amounts of data. Our inspiration came from the challenges we faced in building relevancy-optimized retrieval for our first product, a tool for litigation attorneys.

We are incredibly proud of what we've accomplished. The numbers speak for themselves: over 150M search queries, 2.6M AI conversations, and a community of over 5,400 active users.

Mintlify was one of Trieve's earliest adopters, and we have always been impressed with the quality of their documentation. Their search experience, powered by our technology, set a standard for excellence.

Mintlify's vision extends our own. They are dedicated to empowering builders, and we are excited to contribute to that mission. Denzell and I will be joining the Mintlify team to continue creating powerful tools for developers.

What does this mean for existing Trieve users?

The Trieve Cloud service will be sunset on November 1st, 2025. We are committed to supporting our users through this transition and will assist them in migrating to other platforms or to a self-hosted version of Trieve. All data will be exportable via the API, and we will be available to assist throughout the migration process.

The open-source Trieve infrastructure will continue to be available and maintained. We will carry on managing the codebase and welcome contributions from the community.

For any questions or assistance, please join us on Discord or email us at support@trieve.com.

Open Source Considerations

From the outset, we were committed to ensuring that Trieve could operate without reliance on third-party services. Our goal was to create a sustainable open-source project that would remain accessible and usable, independent of the commercial entity behind it.

While Trieve was a commercial endeavor, we chose the Business Source License (BSL) to protect our users from sudden, restrictive license changes. As part of this acquisition, we are moving to the MIT license to fully embrace the open-source model, ensuring the code can be used by anyone for any purpose.

The repository is available at github.com/devflowinc/trieve, and we encourage contributions and the involvement of new maintainers from the community.

Looking Forward

We are excited about the future and the opportunity to build great things at Mintlify. Please do not hesitate to reach out with questions or for assistance with the migration. Our priority is to ensure a smooth transition for all users.

Mintlify is hiring! If you are interested in joining us, please visit their careers page.

So long, and thanks for all the searches! 🚀

- The Trieve Team

Guide for Self-Hosting Trieve on a VPS

Marcin Stankiewicz — Thu, 12 Sep 2024 11:53:00 GMT

1. Introduction

This guide provides comprehensive instructions for self-hosting Trieve. By following these steps, you'll be able to set up your own Trieve instance on a Hetzner Cloud server.

<Warning> Due to this guide being CPU-only, semantic search and ingest will be SLOW, 2+s for semantic search and ~10 chunks/s on ingest. Fulltext SPLADE and bm25 search types will remain fast, but be aware that running the embedding servers on GPU's is required for a more latency sensitive setup. See our AWS or GCP guides for more information or contact us at humans@trieve.ai for more information. </Warning>

2. Creating the Server

2.1 Prerequisites

Before beginning the server setup, ensure you have:

A domain name with access to its DNS configuration
A Hetzner Cloud account

2.2 Server Setup Steps

Log in to the Hetzner Cloud Console at https://console.hetzner.cloud
Create a new project for your Trieve instance.
Create a public IP address:
- Choose the same location as your intended server location.

![create primary hetzner IP](https://cdn.trieve.ai/blog/self-hosting-guide/hetzner-2.2-image.webp)

Configure your domain:
- Add an A record pointing to the public IP address you just created.

![DNS record setup](https://cdn.trieve.ai/blog/self-hosting-guide/self-hosting-dns-records.webp)

Host	Type	IP
api	A	HETZNER-PUBLIC-IP
auth	A	HETZNER-PUBLIC-IP
dashboard	A	HETZNER-PUBLIC-IP
chat	A	HETZNER-PUBLIC-IP
search	A	HETZNER-PUBLIC-IP
analytics	A	HETZNER-PUBLIC-IP

Add an SSH key to your Hetzner account:
- To view your existing public key in terminal on your local machine, use:
```
cat ~/.ssh/id_ed25519.pub
```
- If you don't have an SSH key, generate one with:
```
ssh-keygen -t ed25519
```

![ssh keygen screenshot](https://cdn.trieve.ai/blog/self-hosting-guide/ssh-keygen.webp)

Go to the "Security" tab in Hetzner Cloud website.
Click on "Add SSH key" and paste your public key.

![hetzner adding ssh key](https://cdn.trieve.ai/blog/self-hosting-guide/hetzner-ssh-key.webp)

For detailed instructions, refer to Hetzner's community guides for Linux and Windows:

Linux: Setting up an SSH key Windows: Generate SSH key using PuTTYgen

Create a new network:
- Go to the "Networks" tab in Hetzner Cloud website.
- Add a new network with the IP range 192.168.1.0/24.

![hetzner create network page](https://cdn.trieve.ai/blog/self-hosting-guide/hetzner-network-setup.webp)

Create a new server:
- Go to the "Servers" tab and click "Add Server".
- Choose the same location as your public IP address.
- Select Ubuntu 24.04 as the operating system.
- Choose a server size (minimum 8vCPU/16GB-RAM, recommended 8vCPU/32GB-RAM).
- In networking settings:
  - Select your public IP.
  - Check "Private networks" and select the network you created.
  - Uncheck public IPv6.
- Select your SSH key.
- In the Cloud config section, paste the provided configuration.
- Replace {PASTE HERE YOUR SSH KEY} with your actual SSH public key.

#cloud-config
users:
  - name: trieve
    groups: users, admin, docker
    sudo: ALL=(ALL) NOPASSWD:ALL
    shell: /bin/bash
    ssh_authorized_keys:
      - {PASTE HERE YOU SSH KEY}
packages:
  - fail2ban
  - firewalld
  - jq
  - caddy
  - make
package_update: true
package_upgrade: true
write_files:
  - path: /etc/docker/daemon.json
    content: |
      {
        "ipv6": false,
        "iptables": false
      }
    permissions: '0644'
  - path: /etc/fail2ban/jail.local
    content: |
      [sshd]
      enabled = true
      banaction = iptables-multiport
    permissions: '0644'
  - path: /etc/ssh/sshd_config
    content: |
         Protocol 2
         HostKey /etc/ssh/ssh_host_rsa_key
         HostKey /etc/ssh/ssh_host_ecdsa_key
         HostKey /etc/ssh/ssh_host_ed25519_key
         KbdInteractiveAuthentication no
         UsePrivilegeSeparation yes
         KeyRegenerationInterval 3600
         ServerKeyBits 4096
         SyslogFacility AUTH
         LogLevel VERBOSE
         LoginGraceTime 60
         PermitRootLogin no
         StrictModes yes
         PubkeyAuthentication yes
         IgnoreRhosts yes
         HostbasedAuthentication no
         PermitEmptyPasswords no
         ChallengeResponseAuthentication no
         PasswordAuthentication no
         X11Forwarding no
         PrintMotd no
         PrintLastLog yes
         TCPKeepAlive yes
         AcceptEnv LANG LC_*
         Subsystem sftp /usr/lib/openssh/sftp-server
         UsePAM yes
         MaxAuthTries 3
         AuthenticationMethods publickey
         KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256
         Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
         MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com
         ClientAliveInterval 300
         ClientAliveCountMax 2
         AllowAgentForwarding yes
         AllowTcpForwarding no
         PermitUserEnvironment no
         AllowUsers trieve
runcmd:
  - curl -fsSL https://get.docker.com | sh
  - newgrp docker
  - systemctl enable fail2ban
  - systemctl restart docker
  - firewall-cmd --zone=public --add-masquerade --permanent
  - firewall-cmd --permanent --zone=trusted --add-interface=docker0
  - firewall-cmd --reload
  - firewall-cmd --zone=public --add-port=80/tcp --permanent
  - firewall-cmd --zone=public --add-port=443/tcp --permanent
  - firewall-cmd --zone=public --add-port=22/tcp --permanent
  - firewall-cmd --permanent --new-zone=docker
  - firewall-cmd --permanent --zone=docker --add-source=172.16.0.0/12
  - firewall-cmd --permanent --zone=docker --set-target=ACCEPT
  - iptables -I DOCKER-USER -i docker0 ! -o docker0 -j DROP
  - mkdir -p /etc/iptables/
  - iptables-save | tee /etc/iptables/rules.v4
  - firewall-cmd --reload
  - systemctl restart docker
  - su - trieve -c "git clone https://github.com/devflowinc/trieve.git"
  - sed -i 's/KC_HOSTNAME=\"localhost\"/KC_HOSTNAME=\"\"/' /home/trieve/trieve/.env.example
  - su - trieve -c "cd trieve && cp .env.example .env && docker compose up -d && sleep 3 && docker compose down"
  - reboot

![creating hetzner server part 1](https://cdn.trieve.ai/blog/self-hosting-guide/hetzner-create-server-1.webp) ![creating hetzner server part 2](https://cdn.trieve.ai/blog/self-hosting-guide/hetzner-create-server-2.webp) ![creating hetzner server part 3](https://cdn.trieve.ai/blog/self-hosting-guide/hetzner-create-server-3.webp)

Create the server and wait for the initialization process to complete (approximately 3 minutes). If you want to see the logs during cloud-init initialization, log into the server via SSH and execute:
```
sudo -s
journalctl -f
```
Connect to your server via SSH:
```
ssh trieve@YOUR_SERVER_IP
```

![ssh to see formation logs](https://cdn.trieve.ai/blog/self-hosting-guide/hetzner-formation-monitoring.webp)

3. Trieve Configuration on the Server

3.1 Caddy Configuration

Log in as root:
```
sudo -s
```
Clear the default Caddy configuration:
```
echo > /etc/caddy/Caddyfile
```

Edit the Caddyfile:

nano /etc/caddy/Caddyfile

Replace YOUR-DOMAIN.COM with your actual domain in the following configuration:

dashboard.YOUR-DOMAIN.COM {
    reverse_proxy localhost:5173
}
search.YOUR-DOMAIN.COM {
    reverse_proxy localhost:5174
}
chat.YOUR-DOMAIN.COM {
    reverse_proxy localhost:5175
}
analytics.YOUR-DOMAIN.COM {
    reverse_proxy localhost:5176
}
api.YOUR-DOMAIN.COM {
    reverse_proxy localhost:8090
}
auth.YOUR-DOMAIN.COM {
    reverse_proxy localhost:8080
}

Verify the Caddy configuration:
```
caddy fmt /etc/caddy/Caddyfile
```
Check your DNS A records, should be same as YOUR_SERVER_IP (Hetzner server public ip):
```
ping api.YOUR-DOMAIN.COM
```
If they indicate a different IP address:
- verify the correct configuration on the domain provider's side
- restart the server again (This ensures that the server picks up the latest DNS changes)
Reload Caddy:
```
systemctl reload caddy.service
```

Verify certificate creation:

journalctl -u caddy | grep "successfully"

![verify caddyfile content](https://cdn.trieve.ai/blog/self-hosting-guide/hetzner-certificate-generation.webp)

3.2 Environment Variables Configuration

Switch to the trieve user and navigate to the Trieve directory:
```
su - trieve
cd /home/trieve/trieve
```
Edit the .env file:
```
nano .env
```

Modify the following variables or add them to the end of the file. Replace YOUR-DOMAIN.COM with actual domain name in all the listed environment variables:

KC_HOSTNAME="auth.YOUR-DOMAIN.COM"
KC_PROXY=edge
VITE_API_HOST=https://api.YOUR-DOMAIN.COM/api
VITE_SEARCH_UI_URL=https://search.YOUR-DOMAIN.COM
VITE_CHAT_UI_URL=https://chat.YOUR-DOMAIN.COM
VITE_ANALYTICS_UI_URL=https://analytics.YOUR-DOMAIN.COM
VITE_DASHBOARD_URL=https://dashboard.YOUR-DOMAIN.COM
OIDC_AUTH_REDIRECT_URL="https://auth.YOUR-DOMAIN.COM/realms/trieve/protocol/openid-connect/auth"
OIDC_ISSUER_URL="https://auth.YOUR-DOMAIN.COM/realms/trieve"
BASE_SERVER_URL="https://api.YOUR-DOMAIN.COM"

![trieve env setup](https://cdn.trieve.ai/blog/self-hosting-guide/trieve-env-config.webp)

3.3 Running Trieve

Start the Trieve application:

docker compose -f docker-compose-cpu-embeddings.yml up -d
docker compose up -d

Monitor the logs during startup:
```
docker compose logs -f
```
If everything is well configured, server show more or less such logs:

![trieve logs view](https://cdn.trieve.ai/blog/self-hosting-guide/running-trieve-logs.webp)

4. Keycloak Configuration

Access the Keycloak admin console at https://auth.YOUR-DOMAIN.COM/admin/master/console/#/trieve/clients/list
Log in with the default credentials:
- Username: admin
- Password: aintsecure

Select the "vault" client and add the following configurations:

Valid redirect and Valid post logout redirect URIs:

https://api.YOUR-DOMAIN.COM/*
https://dashboard.YOUR-DOMAIN.COM/*
https://chat.YOUR-DOMAIN.COM/*
https://search.YOUR-DOMAIN.COM/*
https://analytics.YOUR-DOMAIN.COM/*

Web origins:

+
http://localhost:8090

![keycloak config preview](https://cdn.trieve.ai/blog/self-hosting-guide/keycloak-config-preview.webp)

Navigate to https://dashboard.YOUR-DOMAIN.COM/ and create a new account when prompted.

5. Changing Default Passwords (OPTIONAL, BUT RECOMMENDED)

When configuring Trieve, it's crucial to change all default passwords to ensure the security of your self-hosted instance. This section will guide you through changing the passwords for various components of the Trieve stack. You can choose whether to do it through a script or manually.

5.1 Changing via script

Export new password:

export NEW_PASSWORD="WRITE HERE YOUR NEW PASSWORD"

Use sed one-liners command:

NEW_PASSWORD="$NEW_PASSWORD" && sed -i 's/^MINIO_ROOT_PASSWORD=.*/MINIO_ROOT_PASSWORD="'"$NEW_PASSWORD"'"/; s/^REDIS_PASSWORD=.*/REDIS_PASSWORD="'"$NEW_PASSWORD"'"/; s|^REDIS_URL=.*|REDIS_URL="redis://:'"$NEW_PASSWORD"'@localhost:6379"|; s|^DATABASE_URL=.*|DATABASE_URL="postgres://postgres:'"$NEW_PASSWORD"'@localhost:5432/trieve"|; s/^SALT=.*/SALT="'"$NEW_PASSWORD"'"/; s/^S3_SECRET_KEY=.*/S3_SECRET_KEY="'"$NEW_PASSWORD"'"/; s/^CLICKHOUSE_PASSWORD=.*/CLICKHOUSE_PASSWORD='"$NEW_PASSWORD"'/' .env && sed -i 's/POSTGRES_PASSWORD:.*/POSTGRES_PASSWORD: '"$NEW_PASSWORD"'/; s/KEYCLOAK_ADMIN_PASSWORD=.*/KEYCLOAK_ADMIN_PASSWORD='"$NEW_PASSWORD"'/; s/KC_DB_PASSWORD=.*/KC_DB_PASSWORD='"$NEW_PASSWORD"'/; s/CLICKHOUSE_PASSWORD=.*/CLICKHOUSE_PASSWORD='"$NEW_PASSWORD"'/' docker-compose.yml

5.2 Changing by hand

5.2.1 Updating the .env File

Navigate to the Trieve directory:
```
cd /home/trieve/trieve
```
Open the .env file for editing:
```
nano .env
```
Update the following variables with strong, unique passwords:
- MINIO_ROOT_PASSWORD: Change from "rootpassword" to a secure password for the Minio root user.
- REDIS_PASSWORD: Replace "thisredispasswordisverysecureandcomplex" with a new, complex password.
- S3_SECRET_KEY: Change "ssssssssssssssssssssTTTTTTTTTTTTTTTTTTTT" to a new secret key.
- SALT: Change "goodsaltisveryyummy" to a new random string.
- CLICKHOUSE_PASSWORD: Change from "password" to a secure password.
- REDIS_URL: Replace "thisredispasswordisverysecureandcomplex" from URL with a new, complex password.
- DATABASE_URL: Change from "password" in URL to a secure password.
Save the file and exit the editor.

5.2.2 Updating Docker Compose Configuration

Open the docker-compose.yml file:
```
nano docker-compose.yml
```
Update the following passwords in the file:
- For the db service, change the POSTGRES_PASSWORD from "password" to a new, secure password.
- For the keycloak service, change the KEYCLOAK_ADMIN_PASSWORD from "aintsecure" and KC_DB_PASSWORD from "password" to a new, secure password.
- For the keycloak-db service, change the POSTGRES_PASSWORD from "password" to a new, secure password (use the same password as for the db service).
- For the clickhouse-db service, change the CLICKHOUSE_PASSWORD from "password" to a new, secure password.
Save the file and exit the editor.

5.3 Applying the Changes

After updating the passwords, you need to restart the Docker containers to apply the changes:

Stop the running containers, and remove volumes with old passwords:
```
make clean
```
Start the containers with the new configuration:
```
docker compose up -d --force-recreate
```
After changing all passwords, verify that all services are running correctly:
```
docker compose ps
```
Test the Trieve application to ensure everything is functioning as expected.

Remember to store these new passwords securely, such as in a password manager. Never share them or expose them in public repositories or logs.

6. Troubleshooting

If you're unable to access the Keycloak admin panel due to SSL certificate issues, which can occur when the SSL certificates haven't been properly generated or applied, you can use the following workaround:

Temporarily comment out the KC_HOSTNAME variable in the .env file:

sed -i 's/^KC\_HOSTNAME/#KC\_HOSTNAME/g' .env
docker compose up -d --force-recreate

Also temporarily change the SSH settings in the file /etc/ssh/sshd_config

sudo sed -i 's/AllowTcpForwarding no/AllowTcpForwarding yes/g' /etc/ssh/sshd_config

Set up an SSH tunnel to securely access your server:
```
ssh -vv -D 1337 -C -N trieve@YOUR_SERVER_IP
```
Configure your browser to use a SOCKS5 proxy:
- Host: localhost
- Port: 1337
Access Keycloak via http://192.168.1.2:8080 and complete the configuration. This allows you to access Keycloak without SSL, enabling you to make necessary changes such as disabling the SSL requirement or updating redirect URIs.

After making the necessary changes, restore the KC_HOSTNAME and AllowTcpForwarding variable:

sed -i 's/^#KC\_HOSTNAME/KC\_HOSTNAME/g' .env
docker compose up -d --force-recreate

sudo sed -i 's/AllowTcpForwarding yes/AllowTcpForwarding no/g' /etc/ssh/sshd_config

Remove the SOCKS5 proxy configuration from your browser.

This workaround should only be used temporarily to resolve initial setup issues. Ensure that you properly configure SSL for production use to maintain security.

7. FAQ

Q: What are the minimum server requirements for running Trieve? A: The minimum recommended server is 8vCPU/16GB-RAM, while the optimal server is 8vCPU/32GB-RAM.

Q: How do I update Trieve after installation? A: To update Trieve, pull the latest Docker images and restart the containers. Specific update instructions may vary depending on the version, so consult the official documentation for the most up-to-date process.

Q: Is it possible to use a custom SSL certificate instead of Let's Encrypt? A: Yes, you can use a custom SSL certificate. You'll need to modify the Caddy configuration to use your custom certificate instead of the automatic Let's Encrypt provisioning.

Q: How can I backup my Trieve instance? A: To backup your Trieve instance, you should regularly backup the Docker volumes containing your data and the .env file containing your configuration. Consider using tools like restic or duplicity for automated backups.

Q: What should I do if I forget the Keycloak admin password? A: If you forget the Keycloak admin password, you can reset it by accessing the Keycloak container and using the built-in admin CLI. Consult the Keycloak documentation for specific instructions on resetting the admin password.

Free Alternative to Algolia Docsearch with AI Chat

Nick Khami — Thu, 31 Oct 2024 00:00:00 GMT

Introducing Trieve Vector Inference

Trieve Vector Inference (TVI), our solution for fast, unmetered embedding vector inference in your own cloud or on your own hardware, is now generally available as a standalone product!

Building AI features at scale exposes two critical limitations of cloud embedding APIs: high latency and rate limits. Modern AI applications require better infrastructure.

So, just how good is TVI?

To start, here are our benchmarks:

Additionally, check out the failed requests. Notice that we ran 30,000 with no failures. Without TVI, you’re only making ~7,000. If you’re going through Sagemaker, it’s around ~3,000.

TVI is a simple solution that solves two problems

Rate limits

Multiple API keys rotated on a schedule
Distributed rate limit tracking across microservices
Complex retry logic with exponential backoff
Request queuing systems that rival Kafka in complexity

None of this complexity adds value to your product. It’s pure overhead, stealing engineering time from features that actually matter to your users.

High Latency

High latency robs your magic. When every embedding request takes 300ms+ round trip, your real-time features aren’t really real-time anymore.

The cascading effects are brutal:

Search results that lag behind user typing
Recommendations that feel disconnected from user actions
Chat experiences with noticeable “thinking” delays
Batch processes that take hours instead of minutes

Add in retries for rate limits and your “real-time” feature is now consistently 1+ second behind.

We hope you like the TVI DX

Getting Started

Our deployment process is streamlined for both paths:

AWS: Deployment via Helm through Marketplace
Self-hosted: Docker images and clear documentation for any environment

We back TVI with a 15-day integration guarantee - if you can’t get it running in your environment, we’ll refund your fees. We’ve had teams go from zero to production in under an hour.

The Future of Vector Inference

We’re committed to making vector inference more accessible, faster, and more cost-effective for teams building AI features. Our roadmap includes:

Enhanced monitoring and observability (custom Grafana dashboards coming soon)
Support for more cloud providers (GCP and Azure coming soon)
Additional model optimizations (including quantization and pruning)
Advanced scaling features (automatic horizontal scaling based on load)

Plus, we’re working on some exciting features we can’t talk about yet. If you think today’s latency numbers are good, stay tuned!

Introducing TVI: Embedding and Reranking Infra Built for Kube

Introducing Trieve Vector Inference

Trieve Vector Inference (TVI), our solution for fast, unmetered embedding vector inference in your own cloud or on your own hardware, is now generally available as a standalone product!

Building AI features at scale exposes two critical limitations of cloud embedding APIs: high latency and rate limits. Modern AI applications require better infrastructure.

So, just how good is TVI?

To start, here are our benchmarks:

Additionally, check out the failed requests. Notice that we ran 30,000 with no failures. Without TVI, you’re only making ~7,000. If you’re going through Sagemaker, it’s around ~3,000.

TVI is a simple solution that solves two problems

Rate limits

Multiple API keys rotated on a schedule
Distributed rate limit tracking across microservices
Complex retry logic with exponential backoff
Request queuing systems that rival Kafka in complexity

None of this complexity adds value to your product. It’s pure overhead, stealing engineering time from features that actually matter to your users.

High Latency

High latency robs your magic. When every embedding request takes 300ms+ round trip, your real-time features aren’t really real-time anymore.

The cascading effects are brutal:

Search results that lag behind user typing
Recommendations that feel disconnected from user actions
Chat experiences with noticeable “thinking” delays
Batch processes that take hours instead of minutes

Add in retries for rate limits and your “real-time” feature is now consistently 1+ second behind.

We hope you like the TVI DX

Getting Started

Our deployment process is streamlined for both paths:

AWS: Deployment via Helm through Marketplace
Self-hosted: Docker images and clear documentation for any environment

We back TVI with a 15-day integration guarantee - if you can’t get it running in your environment, we’ll refund your fees. We’ve had teams go from zero to production in under an hour.

The Future of Vector Inference

We’re committed to making vector inference more accessible, faster, and more cost-effective for teams building AI features. Our roadmap includes:

Enhanced monitoring and observability (custom Grafana dashboards coming soon)
Support for more cloud providers (GCP and Azure coming soon)
Additional model optimizations (including quantization and pruning)
Advanced scaling features (automatic horizontal scaling based on load)

Plus, we’re working on some exciting features we can’t talk about yet. If you think today’s latency numbers are good, stay tuned!

Trieve's New Usage-Based Pricing

Nick Khami — Wed, 02 Apr 2025 10:31:00 GMT

Our New Usage-Based Pricing

Trieve is growing. While effective at first, the tier-based pricing strategy was causing pain for our customers and the team. We’ve also expanded our free tier to make it more generous and builder-friendly.

Restrictive and Large Pricing Tiers

Tiering was designed to control concerns about our largest cost centers: Chunks Stored, File Storage, AI Messages, and Datasets. However, the floors, ceilings, and gaps between our tiers proved to be restrictive for our customer base.

As dataset count hockey-sticks, companies attempting to scale especially those using multi-tenant setups like Conduit and Vapi saw friction managing their usage to avoid jumping into the next tier by exceeding chunk, message, or dataset limits.

New Features that original pricing didn’t include

Trieve has grown so fast in the past year that our pricing table wasn’t able to keep up with the full breadth and depth of our product. Users found these features already baked in as we launched them and the tier-based pricing worked well enough to limit on chunk and dataset count.

For example

We added a first-party integration to chunkr, and our very own VLM-based pdf2md service.
We maintain our own webcrawler called firecrawl-simple for seamless and performant web-scraping driectly into a Trieve search index.
We created a search component generator to prototype and create drop-in discovery components.
We turbocharged our analytics dashboard and routes, added voice search, and the list goes on.

Pricing Table

Without further ado, our pricing table:

Product	Free Tier	Cost
Users	First 5 Users free	$5 / User
Platform Fee		$ 5 / mo
	Storage Cost
Chunk Storage	1000 Chunks (11 MB)	$132 / 1M chunks ($12.07 / GB)
File Storage	10 GB	$0.046 / GB
Datasets	2 datasets	$0.05 / dataset
	Ingestion (Resets at end of month)
Write Tokens	First 3M tokens / mo free	$0.028 / 1M tokens
File OCR	First 100 / mo free	$0.01 / Page
Web Crawling	First 10 pages / mo free	$0.00086 / Page Crawled
Bytes Ingested	First 1 GB	$2 / GB ingested
	Per Search charges
Search Tokens	First 3M tokens / mo free	$0.028 / 1M tokens
Message Tokens	First 263,000 tokens / mo free	$3.528 / 1M tokens
Analytic Events	First 1M events / mo free	$0.0001 / event

Pricing Breakdown

Platform Fee

To account for the drastic price decreases, we feel a base platform fee for a full all in one solution is now needed. This also gives us flexiblity to expand to multiple tiers in the future for even better bulk cost savings in the future.

Chunk Storage

Chunk storage is the largest cost center. We hold all the vectors, payload indices, and metadata in RAM with 2x replication. Holding all of these in RAM allows us to provide the lowest possible latency for all of our searches.

We measure the size of a single chunk in Trieve to be (1536 * 4 bytes/vector) + (256 * 4 bytes/sparse vector) + (4096 bytes/payload ) = 11,264 bytes.

We host n4-standard-64 machines and use hyperdisk-balanced NVMe SSDs for our vector db storage layer. The cost per GB is $8.65/GB of RAM + 0.2/GB = $8.67/GB. We charge a ~40% markup which comes out to $12.07 / GB.

File Storage

S3 charges $0.023/GB. We charge $0.046/GB which is a 100% upcharge on S3 storage.

Datasets

We charge $0.05/dataset. Every dataset created adds a new index into our vector db’s HNSW index. Each dataset also gets an ingestion queue for files. Crawls and chunks have equal priority.

Search and Write Tokens

OpenAI’s text-embedding-3-small charges $0.02/M tokens. We charge a 40% markup to embed the input so that’s $0.02 * 40% markup = $0.028/M tokens. Both searches and writes are charged the same fee.

File OCR

Past the free first 100 pages, we charge $0.01/page. PDF pages average 800 tokens.

For our pdf2md OCR ingestion using gpt-4o, our raw cost is 800 tokens * (0.00001 LLM cost/token) = $0.008/page. Accounting for our markup, it comes out to $0.01 / page.

Web Crawling

We price at the industry standard level which is around $86/100,000 pages or $0.00086/page.

Bytes Ingested

The first 1GB of bytes ingested are free. Overages at the end of the month are $2/GB. This is to account for networking costs and match other databases’ write charges.

Message Tokens

As a basis, we charge per token. We allow proxying OpenRouter or OpenAI through Trieve. Embedding and search steps are bundled into the cost of an AI message in Trieve.

For each message, we create an embeddding and conduct an LLM inference step. OpenAI’s gtp-4o-mini model charges us $0.6/1M tokens and text-embedding-3-small charges $0.02/1M tokens.

Analytics

Trieve provides analytics for all searches and chat messages automatically. The first 1,000,000 searches and messages a month are free and have a 5 year retention period. Any additional events are $0.0001/event.

Trieve Enterprise

The prices above are for our public cloud service.

We offer dedicated cloud services that come with SLA guarantees and all of the enterprise goodies. If you require a custom quote for a large scale usecase, please contact us. We are happy to provide bulk discount quotes for massive usage.

Announcing Trieve's New TypeScript SDK!

nikkitaFTW (Sara V) — Tue, 10 Sep 2024 13:38:00 GMT

If you have used Trieve in a JavaScript application, you probably know that you need to make most of your calls to Trieve using fetch. While this approach is good, it's not ideal, and we want to provide users with an easier way to use our APIs.

Well, behind the scenes we have been working on making Trieve easier to use than ever in JavaScript applications and that includes making a new JavaScript SDK that makes it much simpler to integrate Trieve into any application.

First things first, you can install the new trieve-ts-sdk with your favorite package manager:

yarn add trieve-ts-sdk
# or
npm install trieve-ts-sdk
# or
pnpm install trieve-ts-sdk

And now let's see how it works, and let's take a search call as an example.

Before you would need to do something like:

fetch('https://api.trieve.ai/api/chunk/search', {
  method: 'POST',
  headers: {
    'TR-Dataset': 'dc6f3b0d-cf21-412b-9d16-fb7ade090365',
    Authorization: 'tr-********************************',
  },
  body: JSON.stringify({
    query: 'Sonic the Hedgehog',
  }),
});

While this method works well, it's not the cleanest approach. You will need to have the documentation open next to your code editor, as there are no types to assist you in making your function calls.Now, with the new SDK you can call it like so:

import { TrieveSDK } from 'trieve-ts-sdk';

export const trieve = new TrieveSDK({
  apiKey: '<your-api-key>',
  datasetId: '<dataset-to-use>',
});

const results = await trieve.search({
  query: 'Sonic the Hedgehog',
});

With the help of the exported types it's also much easier to create a much more complicated search that includes, for example, filters:

import { TrieveSDK } from 'trieve-ts-sdk';

const results = await trieve.search({
  query: 'Sonic the Hedgehog',
  search_type: 'hybrid',
  filters: {
    must: [
      {
        field: 'meta.rating',
        range: {
          gt: 80,
        },
      },
    ],
    must_not: [
      {
        field: 'metadata.console',
        match: ['gba', 'wii'],
      },
    ],
  },
});

![screenshot of typed Trieve SDK](https://cdn.trieve.ai/blog/we-have-a-new-js-sdk/fully-typed-trieve-ts-sdk.webp)

And it's not just methods for chunks, we have functions for most of our API that you can use, want to stream a RAG completion? We got that:

const reader = await trieve.createMessageReader({
  topic_id: id || currentTopic,
  new_message_content: currentQuestion,
  llm_options: {
    completion_first: true,
  },
});
handleReader(reader);

We also created comprehensive docs so that all these functions are easy for you to find whether you use TypeScript or not.

Okay, the last step is to install it and get to building search and RAG in your application!

Trieve’s Blog

Accurate Hallucination Detection With NER

How We Do It: Smart Use of NER

Why This Is Important:

Benchmark Results

RAGTruth Dataset Performance

Comparison with Vectara

Why This Works

Why It Matters in the Real World

What’s Next: Room to Grow

The Takeaway

Uncle Flaviar: Conversational AI Drives 5-Figure Revenue Lift

About Flaviar

A Discerning AI for a Discerning Audience: How Trieve and Flaviar Teamed Up

In Their Own Words

The Work & Results: Introducing Uncle Flaviar

How to Build Agentic RAG for any PDF in 10 minutes

Introduction

Step 1: Sign Up for Trieve and Set Up Your Dataset

Step 2: Initialize Your Node.js Project and Trieve Client

Step 3: Configure Your Agent's Search Tool

Step 4: Upload and Chunk Your PDF with Chunkr

Step 4: Asking Agentic Questions

Putting It All Together

Conclusion

Build a Hotel Voice Agent with Trieve + Vapi

Hotel Voice Agent Requirements

Trieve Portion

Step 1: Create a Trieve Dataset

Step 2: Trigger a crawl

Vapi Portion

Add your Trieve API Key as a Vapi Provider Credential

Create and copy Trieve api key in your Trieve dashboard at dashboard.trieve.ai

Copy your API key from Trieve

Navigate your Provider Credentials Page in Vapi

Add your Trieve API key to Vapi

Create a Vapi Knowledge Base that connects to Trieve

Save your Dataset ID from the Trieve dashboard

Copy your Vapi secret key from the Vapi dashboard

Make the HTTP API call to create the Knowledge Base in Vapi that connects to Trieve

Setup your Voice agent in the Vapi dashboard

Conclusion

How we Built 300μs Typo Detection for 1.3M Words in Rust

Sample Queries You Can Try

Creating a dictionary of Words and Frequencies

Using a BKTree data structure to identify and correct typos

Writing the Business Logic to Perform Typo Corrections

Pulling from Redis

Identifying English Words

1. Preliminary English Word Identification

2. Affix Analysis

4. BKTree Search for Non-Dictionary Words

5. Correction Selection

Unexpected Benefits

Future Ideas

Building Search For the YC Company Directory With Trieve, Bun, and SolidJS

Introduction

Step 1. Exploratory Data Analysis

1. Each listed company is represented via its own page linked in the search result list

2. The company view offers the data in a neat JSON object as an HTML attribute!

3. Plan out an approach for pulling this JSON object from the DOM for every company

1. Write a script to get the link to every company available in the directory

4. Settle on a language for pulling each company's view from the link and getting the JSON object

Step 2. Sign up For Trieve's Free Cloud To Get an API_KEY and DATASET_ID

Step 3. Use bun to pull the DOM containing the view for each company and extract the data from it

1. Install bun and scaffold an empty project using bun init

2. Use the happy-dom package to headlessly pull data from the View for each company

3. Stitch each companies structured data into an unstructured text blob and send it to Trieve for indexing and storage

Step 4. Build a SolidJS SPA for interacting with the dataset via search

Conclusion

Cheating at Search with LLMs

Introduction

The Problem: Search Can't Understand Intents Like Comparisons

Our Solution: Let the LLM Do the Hard Work

Making It Fast

Going Even Further

Why This Matters

Build Search and RAG for Any Website with Firecrawl and Trieve

Requirements - free API keys:

1. Converting Signoz docs to Markdown Chunks with Firecrawl

1. Install bun and scaffold an empty project using `bun init`

2. Use the `happy-dom` package to headlessly pull data from the View for each company