Data Storage

Data storage in Epsilla manages the underlying stored data records (chunks) from the knowledge base. Users can perform CRUD (Create, Read, Update, Delete) operations on these records, manage tables, and inspect the details of each record. Additionally, users can view the table schema to understand the structure and organization of stored data.

Inspect Data

If you are create a knowledge base from any data source type other than Basic (which means creating knowledge base from scratch), the knowledge_table will already be automatically . You can inspect the processed data (chunks). By default, the first 20 chunks will be visualized when you enter the tab:

Click a record to inspect the data detail:

By default, each record in Epsilla's knowledge base contains the following fields:

ID: A unique UUID to distinguish each record in the knowledge base.
Content: The chunk content as a string.
Filename: The file's name, the webpage's URL, or the identifier in the data source.
Timestamp: The epoch time (in milliseconds) when the record was loaded or updated.
Metadata: Additional information about the chunk, such as DataSourceType (e.g., file), FileType (e.g., pdf), and Pages (a list indicating the pages where this chunk is located, starting at page 1). Note that this metadata differs from the "Meta Data" configured in Advanced Settings.

If Meta Data is configured, additional fields mapped from that meta data will also be shown in the record details.

Use the Records per page setting to adjust how many records are displayed in each batch.

You can navigate between batches using the Prev and Next buttons to scroll through the records.

Semantic Search

You can enter a natural language query in the search box to find the top K most relevant records based on meaning rather than exact keywords; this is known as semantic search.

Click the settings button to open the advanced search setup. By default, the Content field is semantically indexed by Index. If you have added other semantic indexes to the table, you can select a different index for searching.

You can also specify a filter expression for the search:

It will be applied along with the semantic search to further narrow down the results.

Read more about supported filter expression.

Add New Records

Click the Add New Record button at bottom right corner:

And you can insert new records to the knowledge base (following the table schema):

It is recommended to use the data source's automatic processing of records instead of manually inserting records.

Delete Records

Hover over a record with your mouse, then click the Trash Can icon to delete it.

Confirm Delete:

Deleting records is irreversible. It is recommended to rely on the data source's automatic processing of records rather than manually deleting them.

Inspect Table Information

Hover over a table with your mouse, then click the Edit icon to view and inspect the table information.

The first tab displays the table schema, which defines the structure of the table, including the fields, data types, and semantic indices. Read more about table schema.

The second tab displays the APIs that can be used to programmatically manipulate data in the table, allowing for operations such as creating, reading, updating, and deleting records through code:

For each API, three implementation methods are provided: cURL, Python, and JavaScript, allowing users to choose their preferred language or tool for interacting with the table data programmatically:

Create New Table

If you want to define your own knowledge schema, particularly when building from scratch using a Basic type knowledge base, click the Create Table button to create a new table:

The table fields at the "Create Table" stage define the structure and type of data each column will store. You can specify the field names, data types (such as string, integer, or boolean), and any constraints (e.g., Primary Key) to ensure the data is organized correctly and adheres to the intended format:

Read more about the table fields at create table.