I created an open-source ETL to prepare data for RAG

At Q2BSTUDIO, a company specialized in technological development and services, we are always exploring innovative tools that optimize data processing and analysis. Today we want to talk about CocoIndex, an open-source ETL framework designed to prepare data for artificial intelligence applications such as semantic search and RAG (retrieval-augmented generation).

?? Key Features:

• Dataflow-based programming
• Support for custom logic, allowing integration of different chunking methods, embedding generation, and vector storage
• Incremental updates with built-in state management to avoid unnecessary recomputation
• Python SDK with a RUST core

?? How to Get Started:

1. Installation: To begin, install the CocoIndex library in Python:

pip install cocoindex

1. Postgres setup with the pgvector extension: Make sure Docker Compose is installed and then start a Postgres database:

docker compose -f <(curl -L https://raw.githubusercontent.com/cocoindex-io/cocoindex/refs/heads/main/dev/postgres.yaml) up -d

1. Define the indexing flow: Create a flow to index your data. For example:

@cocoindex.flow_def(name='TextEmbedding')
def text_embedding(flow_builder: cocoindex.FlowBuilder, data_scope: cocoindex.DataScope):
    data_scope['documents'] = flow_builder.add_source(cocoindex.sources.LocalFile(path='markdown_files'))
    doc_embeddings = data_scope.add_collector()

    with data_scope['documents'].row() as doc:
        doc['chunks'] = doc['content'].transform(
            cocoindex.functions.SplitRecursively(language='markdown', chunk_size=300, chunk_overlap=100))

        with doc['chunks'].row() as chunk:
            chunk['embedding'] = chunk['text'].transform(
                cocoindex.functions.SentenceTransformerEmbed(model='sentence-transformers/all-MiniLM-L6-v2'))

            doc_embeddings.collect(filename=doc['filename'], location=chunk['location'],
                                   text=chunk['text'], embedding=chunk['embedding'])

    doc_embeddings.export(
        'doc_embeddings',
        cocoindex.storages.Postgres(),
        primary_key_fields=['filename', 'location'],
        vector_index=[('embedding', cocoindex.VectorSimilarityMetric.COSINE_SIMILARITY)])

At Q2BSTUDIO we specialize in custom technological solutions and are committed to implementing innovative tools like CocoIndex to optimize data analysis processes. If you are looking to improve efficiency in data preparation for artificial intelligence, contact us and discover how we can help you in your company's digital transformation.