# Data Storage

Data storage in Epsilla manages the underlying stored data records (chunks) from the knowledge base. Users can perform CRUD (Create, Read, Update, Delete) operations on these records, manage tables, and inspect the details of each record. Additionally, users can view the table schema to understand the structure and organization of stored data.

### **Inspect Data** <a href="#inspect-data" id="inspect-data"></a>

If you are create a knowledge base from any data source type other than **Basic** (which means creating knowledge base from scratch), the `knowledge_table` will already be automatically . You can inspect the processed data (chunks). By default, the first 20 chunks will be visualized when you enter the tab:

<figure><img src="https://2532879721-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FM0ZX7fId7ifK45ldHWEp%2Fuploads%2Fgit-blob-7d94332e10db70b41d587054906b463fb602ddce%2FScreenshot%202024-09-28%20at%2010.03.01%20PM.png?alt=media" alt=""><figcaption></figcaption></figure>

Click a record to inspect the data detail:

<figure><img src="https://2532879721-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FM0ZX7fId7ifK45ldHWEp%2Fuploads%2Fgit-blob-c925632433a6eaccec7bbd620fe728b6bc9d2e13%2FScreenshot%202024-09-28%20at%2010.03.18%20PM.png?alt=media" alt=""><figcaption></figcaption></figure>

By default, each record in Epsilla's knowledge base contains the following fields:

* **ID**: A unique UUID to distinguish each record in the knowledge base.
* **Content**: The chunk content as a string.
* **Filename**: The file's name, the webpage's URL, or the identifier in the data source.
* **Timestamp**: The epoch time (in milliseconds) when the record was loaded or updated.
* **Metadata**: Additional information about the chunk, such as `DataSourceType` (e.g., file), `FileType` (e.g., pdf), and `Pages` (a list indicating the pages where this chunk is located, starting at page 1). Note that this metadata differs from the "Meta Data" configured in Advanced Settings.

If Meta Data is configured, additional fields mapped from that meta data will also be shown in the record details.

<figure><img src="https://2532879721-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FM0ZX7fId7ifK45ldHWEp%2Fuploads%2Fgit-blob-a7d6dc11658e7893e35b4e796d6a963a626ee891%2FScreenshot%202024-09-30%20at%2012.34.55%20AM.png?alt=media" alt="" width="375"><figcaption></figcaption></figure>

Use the **Records per page** setting to adjust how many records are displayed in each batch.

<figure><img src="https://2532879721-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FM0ZX7fId7ifK45ldHWEp%2Fuploads%2Fgit-blob-d4408daeb3fe7fa5c93ec7e41dbc29a8633bf776%2FScreenshot%202024-09-30%20at%2012.39.23%20AM.png?alt=media" alt="" width="115"><figcaption></figcaption></figure>

You can navigate between batches using the **Prev** and **Next** buttons to scroll through the records.

<figure><img src="https://2532879721-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FM0ZX7fId7ifK45ldHWEp%2Fuploads%2Fgit-blob-0b3a90bed60e951b442b7a5e207d752ac1685f50%2FScreenshot%202024-09-30%20at%2012.39.29%20AM.png?alt=media" alt="" width="184"><figcaption></figcaption></figure>

### Semantic Search

You can enter a natural language query in the search box to find the top K most relevant records based on meaning rather than exact keywords; this is known as semantic search.

<figure><img src="https://2532879721-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FM0ZX7fId7ifK45ldHWEp%2Fuploads%2Fgit-blob-38ad9a98ed374a98122d274c98dd24842db03f4a%2FScreenshot%202024-09-30%20at%2012.51.57%20AM.png?alt=media" alt=""><figcaption></figcaption></figure>

Click the settings button to open the advanced search setup. By default, the **Content** field is semantically indexed by **Index**. If you have added other semantic indexes to the table, you can select a different index for searching.

<figure><img src="https://2532879721-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FM0ZX7fId7ifK45ldHWEp%2Fuploads%2Fgit-blob-986a7967292b7efd065f8107e53051fc4801a450%2FScreenshot%202024-09-30%20at%2012.55.05%20AM.png?alt=media" alt="" width="242"><figcaption></figcaption></figure>

You can also specify a filter expression for the search:

<figure><img src="https://2532879721-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FM0ZX7fId7ifK45ldHWEp%2Fuploads%2Fgit-blob-2c3dadfa0e7bb1ac8a169167153cb64af1e5b589%2FScreenshot%202024-09-30%20at%2012.58.42%20AM.png?alt=media" alt="" width="244"><figcaption></figcaption></figure>

It will be applied along with the semantic search to further narrow down the results.

<figure><img src="https://2532879721-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FM0ZX7fId7ifK45ldHWEp%2Fuploads%2Fgit-blob-b4b7aa378148bf6a30b805a3517bf8f1714cd9ac%2FScreenshot%202024-09-30%20at%2012.59.02%20AM.png?alt=media" alt=""><figcaption></figcaption></figure>

Read more about supported [filter expression](https://epsilla-inc.gitbook.io/epsilladb/epsilla-vector-database/user-manual/search-the-top-k-semantically-similar-records#filter-expression).

### Add New Records

Click the Add New Record button at bottom right corner:

<figure><img src="https://2532879721-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FM0ZX7fId7ifK45ldHWEp%2Fuploads%2Fgit-blob-4e99e2b6f10c81a0361d7bd65f03c663561fbf42%2FScreenshot%202024-09-30%20at%201.02.32%20AM.png?alt=media" alt="" width="76"><figcaption></figcaption></figure>

And you can insert new records to the knowledge base (following the table schema):

<figure><img src="https://2532879721-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FM0ZX7fId7ifK45ldHWEp%2Fuploads%2Fgit-blob-4d1f13b99c4ca22eccc8d7f7ef30c3cc9f454b2a%2FScreenshot%202024-09-30%20at%201.03.53%20AM.png?alt=media" alt="" width="375"><figcaption></figcaption></figure>

{% hint style="info" %}
It is recommended to use the data source's automatic processing of records instead of manually inserting records.
{% endhint %}

### Delete Records

Hover over a record with your mouse, then click the **Trash Can** icon to delete it.

<figure><img src="https://2532879721-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FM0ZX7fId7ifK45ldHWEp%2Fuploads%2Fgit-blob-3283b2cea5241c8047ff262b4bfecb1dafc92817%2FScreenshot%202024-09-30%20at%201.07.45%20AM.png?alt=media" alt="" width="314"><figcaption></figcaption></figure>

Confirm Delete:

<figure><img src="https://2532879721-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FM0ZX7fId7ifK45ldHWEp%2Fuploads%2Fgit-blob-9c7178e7c8edf09c463a3e68937ae00e98bef99e%2FScreenshot%202024-09-30%20at%201.07.50%20AM.png?alt=media" alt="" width="375"><figcaption></figcaption></figure>

{% hint style="info" %}
Deleting records is irreversible. It is recommended to rely on the data source's automatic processing of records rather than manually deleting them.
{% endhint %}

### Inspect Table Information

Hover over a table with your mouse, then click the **Edit** icon to view and inspect the table information.

<figure><img src="https://2532879721-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FM0ZX7fId7ifK45ldHWEp%2Fuploads%2Fgit-blob-b323e01d32e0ac52bd73ed7ef77779663634ff75%2FScreenshot%202024-09-30%20at%201.22.07%20AM.png?alt=media" alt="" width="326"><figcaption></figcaption></figure>

The first tab displays the table schema, which defines the structure of the table, including the fields, data types, and semantic indices. Read more about [table schema](https://epsilla-inc.gitbook.io/epsilladb/epsilla-vector-database/user-manual/create-a-new-table#create-a-table-on-epsilla-cloud-portal-ui).

<figure><img src="https://2532879721-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FM0ZX7fId7ifK45ldHWEp%2Fuploads%2Fgit-blob-8a50d846320a17d1756c4fffdd47771a02dcc1c5%2FScreenshot%202024-09-30%20at%201.22.54%20AM.png?alt=media" alt="" width="563"><figcaption></figcaption></figure>

The second tab displays the APIs that can be used to programmatically manipulate data in the table, allowing for operations such as creating, reading, updating, and deleting records through code:

<figure><img src="https://2532879721-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FM0ZX7fId7ifK45ldHWEp%2Fuploads%2Fgit-blob-683482927d1a00b3ebfe8c33c86aac6259a6fb60%2FScreenshot%202024-09-30%20at%201.24.12%20AM.png?alt=media" alt="" width="563"><figcaption></figcaption></figure>

For each API, three implementation methods are provided: **cURL**, **Python**, and **JavaScript**, allowing users to choose their preferred language or tool for interacting with the table data programmatically:

<div><figure><img src="https://2532879721-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FM0ZX7fId7ifK45ldHWEp%2Fuploads%2Fgit-blob-047c64d266649516b171cc6fa3899e920e4ecf0a%2FScreenshot%202024-09-30%20at%201.24.45%20AM.png?alt=media" alt=""><figcaption><p>cURL</p></figcaption></figure> <figure><img src="https://2532879721-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FM0ZX7fId7ifK45ldHWEp%2Fuploads%2Fgit-blob-a3f7199658e1181596b0c2e1d750abd5bb39f84d%2FScreenshot%202024-09-30%20at%201.25.10%20AM.png?alt=media" alt=""><figcaption><p>Python</p></figcaption></figure> <figure><img src="https://2532879721-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FM0ZX7fId7ifK45ldHWEp%2Fuploads%2Fgit-blob-23381ca32f0b49e366d1fb77ed4b656b1d9605a4%2FScreenshot%202024-09-30%20at%201.25.18%20AM.png?alt=media" alt=""><figcaption><p>JavaScript</p></figcaption></figure></div>

### Create New Table

If you want to define your own knowledge schema, particularly when building from scratch using a **Basic** type knowledge base, click the **Create Table** button to create a new table:

<figure><img src="https://2532879721-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FM0ZX7fId7ifK45ldHWEp%2Fuploads%2Fgit-blob-f22a8c4f108586c047497967cccb1558986646fc%2FScreenshot%202024-09-30%20at%201.34.37%20AM.png?alt=media" alt="" width="128"><figcaption></figcaption></figure>

The table fields at the "Create Table" stage define the structure and type of data each column will store. You can specify the field names, data types (such as string, integer, or boolean), and any constraints (e.g., Primary Key) to ensure the data is organized correctly and adheres to the intended format:

<figure><img src="https://2532879721-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FM0ZX7fId7ifK45ldHWEp%2Fuploads%2Fgit-blob-a8b13157622fda74d348900c687dbfefcedbac41%2FScreenshot%202024-09-30%20at%201.34.30%20AM.png?alt=media" alt="" width="563"><figcaption></figcaption></figure>

Read more about the table fields at [create table](https://epsilla-inc.gitbook.io/epsilladb/epsilla-vector-database/user-manual/create-a-new-table#create-a-table-on-epsilla-cloud-portal-ui).

### Delete Table

Hover over a table with your mouse, then click the **Trash Can** icon to delete the table.

<figure><img src="https://2532879721-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FM0ZX7fId7ifK45ldHWEp%2Fuploads%2Fgit-blob-a9dc2868318b4bf142cf1d8b834d27f304c57272%2FScreenshot%202024-09-30%20at%201.39.22%20AM.png?alt=media" alt="" width="257"><figcaption></figcaption></figure>

Confirm Delete:

<figure><img src="https://2532879721-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FM0ZX7fId7ifK45ldHWEp%2Fuploads%2Fgit-blob-b3d1a919e8d20d5fa71e3a8dc2ece8f9f038f1e5%2FScreenshot%202024-09-30%20at%201.34.08%20AM.png?alt=media" alt="" width="375"><figcaption></figcaption></figure>
