Epsilla
HomeDiscordTwitterGithubEmail
  • Welcome
    • Register and Login
    • Explore App Portal
  • Build Your First AI Agent
    • Create a Knowledge Base
    • Set Up Your AI Agent
    • Publish Your AI Agent
  • Knowledge Base
    • Local Files
    • Website
    • Google Drive
    • S3
    • Notion
    • Share Point
    • Google Cloud Storage
    • Azure Blob Storage
    • Confluence
    • Jira
    • Advanced Settings
      • Auto Sync
      • Embedding
      • Data Parsing
      • Data Chunking
      • Hypothetical Questions
      • Webhook
      • Meta Data
    • Data Storage
    • Programmatically Manage Knowledge Bases
  • Application
    • Create New AI Agent
    • Basic Chat Agent Config
    • Basic Smart Search Agent Config
    • Advanced Workflow Customization
    • Publish and Deployment
    • User Engagement Analytics
  • Evaluation
    • Create New Evaluation
    • Run Evaluation
    • Evaluation Run History
  • Integration
  • Team Member Management
  • Project Management
  • Billing Management
  • Release Notes
  • Epsilla Vector Database
    • Overview
    • Quick Start
      • Run with Docker
      • Epsilla Cloud
    • User Manual
      • Connect to a database
      • Create a new table
      • Drop a table
      • Delete a database
      • Insert records
      • Upsert records
      • Search the top K semantically similar records
      • Retrieve records (with filters and pagination)
      • Delete records
      • Performance Tuning
    • Advanced Topics
      • Embeddings
      • Dense vector vs. sparse vector
      • Hybrid Search
    • Integrations
      • OpenAI
      • Mistral AI
      • Jina AI
      • Voyage AI
      • Mixedbread AI
      • Nomic AI
    • Roadmap
Powered by GitBook
On this page
  • Get Knowledge Base Data Source Config Details
  • Create a New S3 Knowledge Base (And Trigger Initial Loading)
  • Update Knowledge Base Config
  • Reprocess Data In Knowledge Base
  • Check Knowledge Base Data Processing Progress
  • Delete Knowledge Base
  1. Knowledge Base

Programmatically Manage Knowledge Bases

PreviousData StorageNextApplication

Last updated 2 months ago

In this section, we introduce the REST APIs for managing knowledge bases. This allows developers to efficiently interact with and update the knowledge base using automation and code.

Get Knowledge Base Data Source Config Details

Request:

curl -X GET https://etl.epsilla.com/api/v1/datasources/<project_id>/pipeline/<datasource_id> \
     -H "X-API-Key: <Project-API-Key>" 

The project_id and datasource_id are identifiers for your project and your knowledge base's data source. You can obtain the project ID and data source ID from the URL:

Response:

{
    "statusCode": 200,
    "message": "Get pipeline successfully.",
    "result": {
        "name": "<knowledge_base_name>",
        "resource_id": "<datasource_id>",
        "updated_at": 1721318300666,
        "created_at": 1721318285687,
        "status": "Active",
        "sync_status": "synchronized", 
        "last_synced": 1721318341060,
        "creator_user_id": "user1@example.com",
        "configs": {                 // Knowledge base data source configs
            "source_type": "file",
            "source_config": {},
            "auto_sync": false,
            "sync_schedule": "",
            "loader_type": "auto",   // Loading config
            "chunk_config": {        // Chunking config
                "type": "sentence",
                "chunk_overlap": 192,
                "chunk_size": 1024
            },
            "embed_model": "openai/text-embedding-3-large",  // Embedding config
            "web_hook": "",
            "meta_file_pattern": "",
            "meta_fields": []
        },
        "files_status": [    // Data processing status
            {
                "entity_type": "FileSyncStatus",
                "sync_status": "synchronized",
                "last_synced": 1721318325641,
                "name": "<FileName>"
            }
        ]
    }
}

Create a New S3 Knowledge Base (And Trigger Initial Loading)

Request

curl -X POST https://etl.epsilla.com/api/v1/datasources/<project_id>/create \
     -H "X-API-Key: <Project-API-Key>" \
     -d {
       "name": "<knowledge_base_name>",
       "source_type":"s3",
       "source_config":{"aws_key_id":"<secret_key>","aws_access_key":"<access_key>","bucket":"<bucket_name>","prefix":"<object_prefix>"},
       "loader_type":"auto",
       "chunk_config":{"type":"sentence","chunk_size":1024,"chunk_overlap":192},
       "embed_model":"openai/text-embedding-3-large",
       "auto_sync": false,
       "web_hook": "<The full URL that will receive the loading job status update>"
     }

Response:

{
    "statusCode": 200,
    "message": "Success to create and execute datasource <datasource_id> for project <project_id>",
    "result": {
        "project_id": <project_id>,
        "datasource": <datasource_id>, // Please keep this ID for your record.
        "status": "Created"
    }
}

Update Knowledge Base Config

Request

curl -X PUT https://etl.epsilla.com/api/v1/datasources/<project_id>/pipeline/update/<datasource_id> \
     -H "X-API-Key: <Project-API-Key>" \
     -d '{"auto_sync": true, "sync_schedule": "15min"}' # Only provide the entries that need to be updated

Response:

{
    "statusCode": 200,
    "message": "Update pipeline successfully.",
    "result": {
        ...   // The knowledge base config
    }
}

Reprocess Data In Knowledge Base

Request

curl -X POST https://etl.epsilla.com/api/v1/datasources/<project_id>/pipeline/<datasource_id> \
     -H "X-API-Key: <Project-API-Key>"

Response:

{
    "statusCode": 200,
    "message": "Start to execute data pipeline successfully.",
    "result": {
        "task_id": "<data_processing_task_id>",
        "pipeline_id": "<datasource_id>"
    }
}

Check Knowledge Base Data Processing Progress

Request:

curl -X GET https://etl.epsilla.com/api/v1/datasources/<project_id>/pipeline/<datasource_id>/status \
     -H "X-API-Key: <Project-API-Key>" 

Response:

// In process
{
    "statusCode": 200,
    "message": "Pipeline still have 1  un-successful sub_tasks.",
    "result": {
        "sub_task_stats": [
            "SUCCESS",
            "STARTED"
        ]
    }
}

// Completed
{
    "statusCode": 200,
    "message": "Get pipeline status successfully.",
    "result": {
        "status": "synchronized" // synchronized
    }
}

Delete Knowledge Base

Request:

curl -X DELETE https://etl.epsilla.com/api/v1/datasources/<project_id>/pipeline/<datasource_id> \
     -H "X-API-Key: <Project-API-Key>" 

Response:

{
    "statusCode": 200,
    "message": "Delete data source successfully."
}

You can find how to get the Project API Key in .

Project Management