Unlock Your Second Brain: Introducing InsightMiner

Published on August 17, 2025

In an age of information overload, we are constantly consuming articles, books, and documents. We highlight, we take notes, but when it comes time to recall a specific piece of wisdom, it’s often buried under a mountain of digital clutter. What if you could turn your personal library into an intelligent, searchable database that not only finds what you’re looking for but also connects ideas and extracts the most important insights?

Here comes InsightMiner, a powerful command-line tool designed to do just that. It’s a local-first, AI-powered system that transforms your collection of Markdown files into a “second brain,” allowing you to ask complex questions and receive concise, relevant answers drawn directly from your own curated content.

What is InsightMiner?

At its core, InsightMiner is a Retrieval-Augmented Generation (RAG) system that runs entirely on your local machine. Unlike cloud-based solutions, it doesn’t require API keys or send your data to third-party services. Your knowledge base remains private and secure.

It’s built for researchers, students, writers, and anyone who wants to derive more value from their reading material. By processing your documents—book highlights, articles, research papers—it creates a sophisticated vector database that understands the meaning behind the words, not just keywords.

How It Works: From Chaos to Clarity

The magic of InsightMiner lies in its two-step process:

Building the Database (python run.py database create): First, the tool reads all your Markdown files from a designated highlights/ folder. It intelligently breaks the text into semantic chunks, generates vector embeddings (numerical representations of the text’s meaning), and stores them in a local ChromaDB database. This process needs to be done only once to index your entire library.
Querying Your Knowledge (python run.py query "your question"): This is where you interact with your second brain. When you ask a question, InsightMiner performs a semantic search to find the most relevant passages from your documents. It then uses a cross-encoder model to re-rank the results for accuracy, extracts the most potent quotes, and finally, generates a summary of the key insights.

The result is a beautifully formatted, easy-to-digest answer that pulls from multiple sources in your library.

An Example in Action

Let’s say you’ve been reading several books on productivity and want to consolidate the key takeaways. Instead of manually sifting through your notes, you can simply ask InsightMiner:

python run.py query "how to be productive"

Here’s a sample of what you might get back:

🔍 Top 5 Results for: 'how to be productive'

📌 Result 1 (Relevance: 1.00)
📚 Source: 18 Minutes
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
To get the right things done, choosing what to ignore is as important as choosing where to focus.
You can be relatively certain that if you decide when and where you’re going to do those things, you’ll actually, reliably and predictably, get them done.
What can you realistically accomplish that will further your focus for the year and allow you to leave at the end of the day feeling that you’ve been productive and successful?
Set your watch, phone, or computer to ring every hour and start the work that’s listed on your calendar.
At the end of your day, shut off your computer and review how the day went,.
Don’t fight yourself to change your behavior in the midst of the wrong environment; just change the environment.

📌 Result 2 (Relevance: 0.97)
📚 Source: 18 Minutes
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
The rule is simple: When you want to do something, focus. When you don’t want to do something, distract.
Distraction, used intentionally, can be an asset.
In contrast to almost everything else in your life, the more you multitask, the worse you are at it. Practice, in this rare case, works against you.
We don’t actually multitask. We switchtask. And it’s inefficient, unproductive, and sometimes even dangerous. Resist the temptation.
Productivity can be achieved only through imperfection.
Stay alert and adapt to changing situations. Keep your eye on the ball, whichever ball that may be.
If you’re having difficulty starting, though, choose your one thing—the one thing that will make the biggest impact.

📌 Result 3 (Relevance: 0.93)
📚 Source: The Time Chunking Method
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
You need to have a reason for doing something in order to make performing that task worth your time and effort.
The less distracted you are, the more productive you’ll be.
If you have superhuman focus in the morning, but the focus of a bored 3-year-old in the afternoon, test the following schedule: Work for 90 minutes Take a 15-minute break Work for 60 minutes Take a 10-minute break Work for 50 minutes Take a 10-minute break.
If laziness is preventing you from using time chunks, that’s a sign that your reasons are not sufficiently compelling to you.

📌 Result 4 (Relevance: 0.75)
📚 Source: Feel-Good Productivity
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
"thinking I was being taught at medical school to questions of happiness,
fulfilment and productivity. So my final exercise brings us full circle, and
involves learning to think about productivity like a scientist. To experiment
with what brings you meaning. And use those experiments to inform the decisions
you make every hour. ‘Alignment experiments’ can help you test theories about
what might bring you closer to alignment in your daytoday decisionmaking. It’s a
process with three stages. First, identify an area of your life where your
actions feel particularly unfulfilling. The results of your eulogy method,
odyssey plan and wheel of life exercises might have helped with this


📌 Result 5 (Relevance: 0.53)
📚 Source: Feel-Good Productivity
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Adopting an experimental mindset. All scientific experiments have an ‘independent variable’, the one thing you change to see what effect it could have. If you were to change one – just one – independent variable in your life, what would it be? And what effect do you think it would have on your situation?
My life has changed. These days, I know that productivity isn’t about discipline; it’s about doing more of what makes you feel happier, less stressed, more energised.
If you can tap in to what makes you feel most energised and alive, you can get anywhere.

════════════════════════════════════════════════════════════════════════════════
                                  ✨ Insights ✨                                  
════════════════════════════════════════════════════════════════════════════════
✨ Top Insights About 'how to be productive' ✨

1. The less distracted you are, the more productive you’ll be.
2. What can you realistically accomplish that will further your focus for the year and allow you to leave at the end of the day feeling that you’ve been productive and successful?
3. My life has changed. These days, I know that productivity isn’t about discipline; it’s about doing more of what makes you feel happier, less stressed, more energised.
4. Productivity can be achieved only through imperfection.
5. You need to have a reason for doing something in order to make performing that task worth your time and effort.

The tool doesn’t just give you a list of documents; it synthesizes the core ideas into actionable insights. This is the true power of InsightMiner—it transforms passive notes into active, usable knowledge.

Why Go Local?

In a world dominated by cloud services, the decision to keep InsightMiner fully local is a deliberate one. It offers several key advantages:

Privacy: Your data never leaves your machine.
No Costs: You are not dependent on paid APIs.
Offline Access: Your knowledge base is always available.
Customization: You have the freedom to swap out the underlying AI models to fit your specific needs.

If you’re ready to take control of your digital notes and build a truly personal knowledge engine, give InsightMiner a try. It’s a step towards working smarter, not harder, and unlocking the collective wisdom hidden in your own reading.