Supported Formats & Limits

This page covers the technical details of how thinnestAI processes your knowledge sources — supported formats, size constraints, chunking strategy, and tips for getting the best results.

File Format Reference

Source Type Limits

Knowledge Base Limits

Chunking and Processing

When you add a source, thinnestAI breaks the content into smaller pieces called chunks. This is essential for accurate retrieval — the agent finds the specific chunk that answers the question, rather than searching through an entire document.

How Chunking Works

1. Content extraction — Text is extracted from the source format (PDF parsing, HTML scraping, etc.).
2. Cleaning — Headers, footers, navigation elements, and other noise are removed.
3. Splitting — Content is split into chunks at natural boundaries:
   • Headings and section breaks
   • Paragraph boundaries
   • Sentence boundaries (for very long paragraphs)
4. Overlap — Adjacent chunks share a small overlap to preserve context across boundaries.

Chunk Parameters

Why Chunk Size Matters

• Too large — Chunks contain mixed topics, making retrieval less precise.
• Too small — Chunks lack context, making answers less comprehensive.
• Just right — Each chunk covers one topic or concept completely.

The default settings work well for most content. The system automatically adapts to your document structure.

Embedding Models

thinnestAI converts each chunk into a vector embedding — a numerical representation that captures the semantic meaning of the text. These embeddings power semantic search.

How Embeddings Work

"What is your return policy?"
        │
        ▼
   Embedding Model
        │
        ▼
[0.023, -0.156, 0.891, ...]  ← 1536-dimensional vector

Similar concepts produce similar vectors, so a search for "refund process" will find chunks about "return policy" even though the words are different.

Embedding Details

Vector Storage

Embeddings are stored in a PostgreSQL database with the pgvector extension, enabling fast similarity search at scale. For larger deployments, thinnestAI also supports LanceDB and ChromaDB as vector storage backends.

Search & Retrieval

When your agent searches the knowledge base, the system runs a hybrid search:

Retrieval Parameters

Best Practices for Knowledge Quality

Document Structure

Good structure produces better chunks and more accurate retrieval.

Do:

• Use clear headings (H1, H2, H3) to organize content.
• Write one topic per section.
• Use bullet points and numbered lists for key information.
• Include a table of contents for long documents.

Don't:

• Put multiple unrelated topics in one section.
• Use images as the primary way to convey information (images aren't indexed).
• Include large blocks of unformatted text without breaks.

Content Quality

• Be specific — "Our PAYG plan charges ₹1.50/min for voice and ₹0.50/message for chat" is better than "See our pricing page for details."
• Be complete — Each section should be self-contained. Don't rely on context from other sections.
• Be current — Remove outdated content. Old information can cause incorrect answers.
• Avoid contradictions — If two sources say different things, the agent may give inconsistent answers.

Organizing Knowledge Bases

Example organization for a SaaS company:

Testing Your Knowledge

After adding sources, test with real questions:

1. Ask direct questions — "What's the refund policy?" Should return exact information.
2. Ask rephrased questions — "Can I get my money back?" Should find the same content.
3. Ask edge cases — "What happens if I cancel after 60 days?" Should find the closest relevant policy.
4. Check for gaps — If the agent says "I don't have information about that," you may need to add more content.

Troubleshooting

Next Steps

• Knowledge Sources — Step-by-step guide for each source type.
• Knowledge Overview — Creating and assigning knowledge bases.