Having some time off between jobs and feeling somewhat outraged by the price, I decided to see if I could have a go at this myself. For context, this is my very first car which I bought earlier this year, and I didn’t know the difference between a wrench and a spanner.

It turned out that this was a common defect, and Mazda had published a technical service bulletin ¹ for technicians, which I found through a Reddit thread ² from another CX-30 owner who had faced the same issue. Having gained enough confidence that this also applied to my car based on the diagnosis and the affected models, I took a very small leap of faith and bought a plastic panel remover and a 10-mm combination spanner on Amazon for a combined total of 13 pounds.

The actual root cause was that one of the three bolts on the liftgate was loose due to a manufacturing error. The fix was as simple as tightening it with a spanner, and then magically, the loud noise was no more.

With no prior experience in fixing cars or even knowing about car parts, this entire process felt like software problems I encounter every day. For a given problem I had, other people had already faced it and shared how they addressed it on the Internet, and I could refer to a public document from the first party to compare the details and find the actual issue that led to the fix.

Some of the hurdles you need to jump over to get to a solution like this are often less financial or technical than mental. After all, all it took was a small spanner to tighten a small bolt. Still, many people would rather pay other people to do it and not worry about this themselves at all. While this is more than reasonable, if you are the type of person who would not hesitate to open a small PR to fix a bug or even a spelling mistake in an open-source software dependency, I would recommend trying to treat a physical problem like this like it’s a software problem. It might give you a sense of satistifaction of problem solving, and better yet, it might give you back some of the money you lost hosting your free software.

The main lesson I learnt from this experience is that you have to solve retrieval first before you solve RAG. To achieve a desired user experience, you need to own (or at least be able to contribute to) the entire pipeline from indexing to retrieval to reranking to generation, rather than simply adding a generation step on top of an existing search system.

In this post, I will describe my personal journey of trying to build a search system for company documents in the UK, and how I ended up writing a Python client for Apache Solr.

Search Pipeline

With the above lesson in mind, I thought about building a sandbox environment where I could tune different search pipeline configurations against different sets of requirements. As a starting point, for example, I’d ask myself:

• How often do I need to ingest new documents?
• How fast should search be?
• What is the distribution of keywords like in the document domain?

Answers to these questions will influence the system architecture, often with tradeoffs across different components.

For instance, the frequency and lengths of new documents inform whether we can apply compute-heavy document understanding or enrichment steps. We cannot spend multiple seconds ingesting a single document if new documents keep arriving while processing it.

If this requirement is not strict, then we can potentially afford to extract metadata from documents such as named entities or rephrasing some of the keywords and setting up specific fields to search over for BM25, which would improve both latency and relevance for sparse retrieval.

Keyword distribution also has consequences on how we handle ambiguity and vocabulary mismatch between the user query space and the document domain, with implications for the choice of the embedding algorithm, vector search databases, and vector search algorithms.

The Side Project

The actual project I wanted to work on was a better search system for company information on UK startups using the Document API from Companies House ¹. This was motivated by my personal experience that startups are generally much less transparent when things are not going well, and that the current user experience of Companies House is not ideal in terms of document discoverability and readability.

Providing a better search system for such information could potentially help alleviate the pain point of not having enough realistic information on company financials and governance structures prior to joining a startup. This assumes the existence of documents disclosing such information in the Companies House database already. This is often the case after startups raise Series A or Series B rounds when they typically accelerate their hiring efforts.

In Companies House, the documents are almost always indexed as PDFs and the frequency of document updates is generally low, say, no more than one document per day per company. This necessitates (1) a document understanding step from PDF to some form of structure that can be indexed into the search system and (2) some flexibility on latency to run this step, opening up the possibility of using something more advanced than simple PDF parsing, such as vision-language model-based SmolDocling.

With those requirements, I came up with the following diagram with necessary components for setting up the search pipeline.

Searching for Search Servers

I started to look for the libraries and technologies to fill in those components in the diagram above, and considered multiple options for the search server. With the exception of Meilisearch which is built in Rust, the mainstream enterprise search options are Elasticsearch, OpenSearch, and Apache Solr, which are all built on Apache Lucene. I had experience using Apache Solr in a previous RAG project and decided to stick with it due to its robust sparse-retrieval performance and ecosystem maturity.

This HackerNews comment from tekkk on Aug 16, 2020 echoes my sentiment on Solr:

Solr is one of those technologies which works but isn’t really glorious to use and is a bit stuffy with its XML configurations and Java interfaces. It’s a bit of a shame, because search engines are so popular nowadays and everybody seems to be fixated on using ElasticSearch, which from what I’ve read and heard is resource-hungry and not really suited for simple text-search.

and note that this comment predates the rise of LLMs which subsequently led to the popularity in RAG and has since invited a number of ML practitioners who are typically not familiar with the Java ecosystem, including myself.

I believe this is the primary point of friction where the interface becomes the main bottleneck for developing a search system. For example, the de facto official Python client for Apache Solr is pysolr, which has been around for well over a decade and is implemented in one 1500-line Python file.

This unfortunately falls short of expectations for a modern Python experience. For instance, pysolr does not support async operations at all, making concurrent queries inefficient compared to libraries that leverage Python’s async features. Indexing documents is also less ergonomic, and there is no built-in type safety, which can lead to runtime errors and make integration with typed codebases more challenging.

Another source of confusion for new users is that pysolr exposes Solr’s query syntax directly. Parameters like q, fl, and qf are somewhat terse abbreviations for a new user, and the relationship between query parameters and query parsers is not clearly documented or enforced by the client. This can make it difficult to understand the hierarchy and structure of queries, especially for those unfamiliar with Solr’s internals.

Let’s take a look at an actual example. The following snippet sets up a Solr client and uses Extended DisMax Parser to find the keyword “preference shares” (prioritizing the summary field matches over the content field) and returns the results grouped by company_name field with a limit of 3 documents per group.

As a prerequisite, Solr’s search request is constructed as HTTP query parameters and is sent as a GET request to a given collection of documents. In pysolr, those are set directly as Python dictionaries as follows.

from pysolr import Solr

solr = Solr('http://localhost:8983/solr/company_documents', timeout=10)

params = {
    'defType': 'edismax',
    'q': 'preference shares',
    'qf': 'summary^2.0 content^1.0',
    'group': 'true',
    'group.field': 'company_name',
    'group.limit': 3,
    'group.ngroups': 'true'
}

results = solr.search(**params)

The response here is an untyped dictionary that the user has to parse and validate manually, and it’s also hard to understand all the fields needed in the params and how they interact with each other.

Writing a New Client

This is the point where I decided to rewrite the Python client from scratch with a modern setup with full type-safety and async support. My aim of building this client was to provide a better developer experience with a clearer interface with full IDE support which also doubles as a wrapper around the official Solr Query Guide ², and having a standardised way to provide the document class in the search responses using Pydantic models.

I named this client Taiyo after the Japanese translation of the word sun which Solr-related projects often get the inspiration from, such as sunburned which is another Python client. The full documentation is available in Taiyo Docs and the source code is available on GitHub.

Going back to the Extended DisMax query example. Taiyo now allows you to set up a query parser like the following:

from taiyo.params import GroupParamsConfig
from taiyo.parsers import ExtendedDisMaxQueryParser

parser = ExtendedDisMaxQueryParser(
        query="preference shares",
        query_fields={"summary": 2.0, "content": 1.0},
        configs=[
            GroupParamsConfig(by="company_name", limit=3, ngroups=True)
        ],
)

with GroupParamsConfig object passed in as a config object and each field autocompleted in the IDE with documentation referenced from the official query guide.

Alternatively, it’s also possible to set up the same exact query with a pandas-like chaining on the parser object:

from taiyo.parsers import ExtendedDisMaxQueryParser

parser = ExtendedDisMaxQueryParser(
    query="preference shares",
    query_fields={"summary": 2.0, "content": 1.0}
).group(
    by="company_name",
    limit=3,
    ngroups=True
)

Whether you use config objects or method chaining is a matter of personal preference or complexity. If you need a highly custom/multiple layers of configuration, you might want to opt for config objects that can be set up in different places (i.e in a iteration loop or loaded from a separate file).

If a potential user is hesitant on switching to a new client, they can also make use of Taiyo’s query parsers which serialises into Python dictionaries that other clients like pysolr accepts.

from taiyo.parsers import StandardParser

parser = StandardParser(
    query="preference shares",
    query_operator="OR",
    filter_queries=["created_at:[2025-01-01 TO *]"],
    rows=10,
)

# Build query parameters as dictionary
params = parser.build()

# {
#     'q': 'preference shares',
#     'q.op': 'OR',
#     'fq': ['created_at:[2025-01-01 TO *]'],
#     'rows': 10,
#     'defType': 'lucene'
# }

# Use with httpx or any HTTP library
import httpx
response = httpx.get(
    "http://localhost:8983/solr/company_documents/select",
    params=params
)

You can also make use of the document model when searching for documents. For instance, if you set up the following CompanyDocument class derived from taiyo.SolrDocument, then you can specify this in the parser such that the responses are fully typed.

from taiyo import StandardParser, SolrDocument

class CompanyDocument(SolrDocument):
    document_id: str
    company_name: str
    company_number: int
    category: str
    created_at: str
    summary: str
    content: str

# Search returns typed results
parser = StandardParser(query="preference shares")
results = client.search(parser, document_model=CompanyDocument)

# Access typed company documents
for doc in results.docs:
    print(f"{doc.category} document for {doc.company_name} ({doc.company_number})")
    print(f"Summary: {doc.summary}")

With the async support with httpx, you can also index typed documents asynchronously as follows:

import asyncio
from taiyo import AsyncSolrClient, SolrDocument

async def ingest(docs: list[CompanyDocument]):
    async with AsyncSolrClient("http://localhost:8983/solr") as client:
        client.set_collection("company_documents")

        # Split into batches and process concurrently
        batch_size = 100
        batches = [docs[i:i + batch_size] for i in range(0, len(docs), batch_size)]

        # Index all batches concurrently
        await asyncio.gather(*[client.add(batch, commit=False) for batch in batches])
        await client.commit()

docs = ...
asyncio.run(ingest(docs))

Making something I want

There are several lines of work planned, including supporting more query parsers such as XML parsers, along with clearer documentation. Also note that this client doesn’t abstract away the Lucene query syntax in the query parameter, which could be provided as a utility function in this library, should I find an intuitive interface.

The inevitable next step is to build out the original side project I set out to work on, which is to build a better full-text search system for company documents. The best thing about this client is that it would not be the end of the world if nobody finds it useful, as long as it helps me on my original journey and I can find value in it myself.

YC’s famous motto Make something people want ³ is great advice for building venture-scale companies, but making something I want is more than enough as a heuristic for building side projects while keeping a full-time job, which was definitely the case for Visprex in 2024 and Taiyo in 2025. I’m looking forward to writing about another side project this year, hopefully more often than in the past two years.

On the technical side, I worked on my data visualisation tool and wrote about it on this blog. On the non-technical side, I took on two new hobbies: long-distance running and chess.

Numbers

While somewhat unrelated at first glance, the common denominator between running and chess is that your progress is measurable and easily trackable, and that you have full accountability over the results. Luck plays a very small role in running and chess compared to team sports or other types of games of skill that I had dabbled in. I signed up on Chess.com and reinstalled Strava on my phone after I accepted the job offer in February, and eventually it felt natural to me that I worked towards a certain goal for each. After all, I liked optimising metrics.

Around 8 months ago, I set myself to run a full marathon in 4 hours ¹ and to reach a 1000 rating on Chess.com ² by the end of the year. As of writing, I have achieved the goal for chess and fell short for running. My Chess.com profile shows I’ve played 865 games of Rapid Chess ³ in the past year with a peak rating of 1082 on the 17th of December, which puts me at around the 85th percentile on the platform. During this span, I won 446 games, drew 33, and lost 386. For running, I’ve recorded 60 runs for a total of 577 kilometers (358 miles) and I completed the Taipei Marathon two weeks ago with a personal net time of 4 hours, 11 minutes and 6 seconds, which is around the 54th percentile for the finishing time for males ⁴.

Approach

Those results are in line with my expectations based on the rates of progression from practices throughout the year, although it was much easier to form these expectations for chess as it had a much faster iteration cycle.

For marathon training, I did one long run every weekend with the distance progressively increasing from 10 km to 31 km, with occasional weekday runs between 5 and 10 kilometers. I initially suffered back pains after long runs, which then slowly went away after I started core strength training with planks. I also discovered r/Marathon_Training a few months before the race, which was helpful for learning fueling strategies and what to expect of the unknown distance, which for me was the last 11 kilometers. I can now confirm the conventional wisdom from this subreddit: a full marathon can be split into two halves - the first 32 km and the last 10 (or the first 20 miles and the last 6.2).

Chess, on the other hand, was much easier to progress and the overall mental burden was lower. After picking up the basics, I have exclusively played variations of The London System as White and King’s Indian Defence as Black, and used Chess.com’s Game Review functionality to look back on my games and watched GothamChess to learn principles and openings. As a side note, chess’s popularity exploded during COVID thanks to a combination of people staying at home and the release of Queen’s Gambit on Netflix, and while the online resources are truly wonderful, there’s lots of drama and gossip entertainment as competitive chess is a unique sport where the world’s top players are chronically online and active in the streaming space.

Am I good enough?

Long-distance running and chess are generally hard hobbies to get into, in that the initial learning curve is steep where lots of people drop off. Like language learning, you feel like you’re venturing into a new world and learning a lot at first, and then there’s a phase of plateau where you have to keep practicing consistently to get into the intermediate level. Once you get past it, however, there are lots of people who are very good, and it’s objectively clear that they’re very good.

The visibility of people who are very good often makes you question why you should pursue it at all, especially if you have a competitive personality like myself. It is obviously unreasonable to think that you can be decent at something you started less than a year ago, but I had to develop a sense of patience to limit the comparison within versions of myself and not with others. Accepting mediocrity, both internally and externally, was a sign of maturity that helped me focus on the trajectory and not the current state.

Eventually, I started to appreciate the non-competitive aspects of these activities. Playing speed chess during odd hours of the day made me better at quickly switching my brain into a problem-solving mode under time pressure, which was a nice continuation from practicing for coding interviews earlier this year.

Running requires very little gear compared to other sports, meaning that there’s basically no constraints on locations. This added another aspect to my travels. Exploring new places on foot makes you feel like you belong in that new environment and allows you to see places a tourist might not. Apart from my usual runs in Victoria Park in London, I also went running in Innsbruck, Munich, Zurich, Naples, Kyoto, and Tokyo, and came back with a better sense of the geography each time.

The year ahead

Next year, I’m planning to get back into basketball where the goal is to play for a local amateur team and get some playing time. Another goal is to do a 5 km run in 20 minutes, which is practically a new sport. Unlike marathons which I can realistically attempt once or twice a year, I can aim for my personal best on a random weekday morning, and I’m excited to see how fast I can go.

Why Small Data?

In February 2023, a blog article titled Big Data is Dead was published by MotherDuck ¹, a cloud data warehouse company built on top of DuckDB. Among several reasons why the prophesied Big Data movement is often misguided, one explanation that resonated with me the most is the separation between the analytics workload and the overall data sizes.

Reflecting on his time working on Google’s BigQuery, the author Jordan Tigani writes:

Customers with giant data sizes almost never queried huge amounts of data

which is also reflective of my experience working at a large marketplace company with hundreds of millions of monthly transactions, where key analytics decisions were made from representative samples by data analysts and domain experts.

Another distinction I would like to illustrate here is that the type and size of data required in analytics and modelling are somewhat different from one another.

In the modern machine learning context, modelling is often associated with building a predictive system (i.e. with neural networks) which benefits from having more training data. More concretely, it helps to train the model with edge case scenarios which minimises the loss function on top of capturing general data trends, as the model is able to run inference on specific data points using high-dimensional features.

On the other hand, the output of analytics work is mainly actionable insights for business functions, where the stakeholders’ primary interest is coming up with a policy or a campaign that is effective for the average user in some segment or cohort. For those purposes, having representative user data is often more than enough.

There are always nuances in how analytics workload should be handled, but I do agree with the main points of the article that it is time to acknowledge the headaches associated with infrastructures built for scale, and that there’s room for small software in some contexts such as analytics and linear modelling where a simple, single-machine software can solve most of your problems at hand.

Building Small Software

In the spirit of building small software, I have been working on a side project called Visprex for over a year now, which enables you to use your browser as a data visualisation tool for your CSV files.

The main features of Visprex include the ability to plot histograms, scatterplots, and a correlation matrix, with additional utility tools for inspecting individual data points from plots, feature transformations, and filtering. See Visprex Docs for more details and use cases.

Motivation and Architectural Choices

The focus on these visualisation steps is motivated by my background in economics which I read for my undergraduate degree. I spent a lot of time thinking about data distributions and linear relationships between features during this time, and I believe a small tool like Visprex can help you quickly build an intuition about your dataset without referring to data visualisation libraries on your software, which in my opinion is the least creative part of data science workflows.

While this software is not suitable for large datasets (>100 MB), it is sufficient to get an idea of data distributions from representative samples in many analytics cases.

Another choice I made for this small data use case is to process everything on the frontend with TypeScript. While this naturally limits the capacity to handle large data, it provides two important benefits for the potential user and the maintainer.

The benefit for the user is data privacy. You can load sensitive data and don’t have to worry about it leaving your laptop. Convincing the user of this, however, is another story. A technical person can check the Network tab to see that there are no network calls made. Another simpler way to ensure this is to turn off your Wi-Fi after the site loads. While some mobile applications are successful at this, communicating that your website works offline remains a difficult problem.

The benefit for the maintainer is low upkeep. It costs very little for me to run this project as everything is bundled into less than 50 kilobytes of static assets and a few hundred kilobytes of minified JavaScript code, and I do not have to worry about maintaining a database which tends to be costly even for a small project.

Upcoming Features

I’m planning on adding a few more features to Visprex, and one is visualisation for time series data. I’m using Papa Parse for parsing data from CSV files, and I need to come up with a better way to handle timestamps of different formats to conditionally display line charts for different features on the Y axis and the timestamp on the X axis.

Another feature I’m keen to work on is DuckDB Wasm integration, which would add a SQL editor tab for transforming and querying data and then updates the in-memory data automatically as inputs for your visualisation. While this increases the bundle size and is useful for advanced users who can write SQL queries in practice, it addresses the main limitation of everything-on-the-frontend approach, as WebAssembly can handle up to 4 GB of in-memory data ².

Looking Back

It’s been an enjoyable and fulfilling experience so far to be able to build and showcase a tool that I personally think is useful to my friends and colleagues without worrying about data privacy or running costs. As my day jobs have always been in backend systems, I learnt a lot about building frontend applications with TypeScript and React along with managing a domain and writing documentation.

I’d also like to give credit to Dr Sean Brocklebank from the University of Edinburgh and Dr Anna Brocklebank for providing feedback on the early iterations of Visprex.

If you have any feedback or feature requests, please feel free to reach out at kengo@hey.com. If you’re interested in how I built Visprex, the code is open-source on GitHub.

Edit: HackerNews

One week after I published this blog post, I made a submission to HackerNews titled Show HN: Visprex – Open-source, in-browser data visualisation tool for CSV files. This unexpectedly went on to the front page over the weekend and I received a number of useful feedback comments, suggestions, upvotes, and GitHub stars. I was particularly happy to see this comment from parsimo2010:

I like this a lot- I am going to show it to my students! They seem to hate learning R, and while this doesn’t prevent them from having to build a model, this will speed up the exploration steps.

which describes the exact audience and the use case I intended Visprex for. Thank you to the HN community if you also came to this blog post from there. I’m planning to work on the features mentioned in the threads (timestamp parsing and data preprocessing), and will keep writing on this blog about major feature releases and updates.