pnpm - Performant Node Package Manager

The Problem

Every application is created to tackle a problem. Whether it's a small issue or a big one, if it doesn't address a specific problem, then it's hard to understand why it was made. pnpm was developed to fix a particular issue caused by the old npm. To illustrate this problem, let me share a meme with you.

node_modules has many files, so it uses a lot of space. Now, imagine you're working on several projects, and each has its own node_modules. It's not entirely different; for example, if two projects use the same version of React for the frontend, you'll have React node_modules in two places, taking up space.

That was just an example; in reality, there may be a lot of packages used in different projects, and every time you run npm install, a network call is made to download all of these again, even though you have one copy in another project. Wouldn’t it be great if we could somehow optimise this? And that’s where pnpm comes in!

The Solution

You might already have an idea of how to solve this problem: instead of downloading a module for every project, why not reuse it? Of course, creating a copy wouldn't work because we would still be losing space. The solution is to place the required module in a centralised location, and all projects that need it use links that point to the main folder containing the module's actual code. This works just like pointers, which store the address and can reference a variable through the address.

How To Achieve This?

pnpm achieves this using hard links and symbolic links.

A hard link is an additional "name" for the same data on the disk. In Linux, every file is actually a link to an inode (the data's address). A hard link just creates a second pointer to that same address. If you delete the original file name, the data is not deleted. The data only disappears once all hard links to it are deleted. It cannot point to directories and cannot cross different partitions (because inode numbers are unique to each drive).

ln original.txt my_duplicate.txt # creating a hard link or original.txt

Even if you delete original.txt, you can still open my_duplicate.txt and see all your data. They are effectively the same file.

A symbolic link is like a "redirect" sign. It contains the text of a path (e.g., ../my-files/photo.jpg). If you delete the original file, the symlink "breaks" because the path it points to is gone. It can point to directories and can cross over to different hard drives/partitions.

ln -s original.txt my_shortcut.txt

If you move original.txt to a different folder, my_shortcut.txt will stop working.

When you run pnpm install express The following steps take place in order

Step 1: The Global Store (The Source)

When you request a package (e.g., express), pnpm first checks its Global Store (usually located at ~/.pnpm-store).

• If the package isn't there, pnpm downloads it once.

• This store is "content-addressable," meaning it saves every unique file based on its content, not its name. express@latest express@4.1.3 are unique, so both of them will be stored separately.

Step 2: Creating the Virtual Store (Hard Links)

Instead of copying express into your project, pnpm creates a hidden folder: node_modules/.pnpm. This is called the Virtual Store.

• The Action: pnpm creates Hard Links from the Global Store into this .pnpm folder.

• The Result: The files now "exist" in your project folder, but they take up zero extra disk space because they point to the same data on the disk as the Global Store.

Step 3: Nesting Dependencies (Symlinks)

Packages often have their own dependencies. For example, express needs body-parser.

• The Action: Inside the .pnpm folder, pnpm creates Symlinks to connect packages to their own dependencies.

• The Result: This creates a massive, nested web of symlinks that satisfies exactly what each package needs to "see" to function.

Step 4: The "Public" node_modules (Symlinks)

Now that the "hidden" .pnpm The folder is ready; pnpm needs to make the packages you actually asked for visible to your code.

• The Action: pnpm creates a Symlink in the root of your node_modules project that points to the package buried inside the .pnpm virtual store.

• Example: node_modules/express —> .pnpm/express@4.18.2/node_modules/express

Step 5: Resolution

When you write import express from 'express' In your code:

1. Node.js looks in node_modules/express.

2. It follows the Symlink into the .pnpm virtual store.

3. It finds the actual files, which are hard-linked to the Global Store.

4. Your app runs perfectly, and your SSD stays empty.

The Problem of Phantom Dependencies

In npm, dependencies are "hoisted" or flattened to the root of node_modules. If you install express and express depend on debug, you could technically import 'debug' use it in your code even if you didn't list it in your package.json.

This looks like a feature, but it can cause some random bugs, if in future express decided to drop debug and use some other package like better-debug Now, suddenly, for some reason, you get an error module not found because you never explicitly installed that package.

Because pnpm uses symlinks to only expose what is explicitly in your package.json, your code cannot access packages it doesn't officially depend on. This makes your builds much more predictable and secure.

Conclusion

This is just two of many benefits of pnpm, if your project follows a monorepo structure, then using pnpm can help you track and manage dependencies in a lot better way compared to npm.

It is significantly faster in automated environments like GitHub Actions or Jenkins. If a package has a "post-install" script (like building some C++ code or generating files), pnpm can cache the result of that script. Next time you install that package (even in a different project), pnpm just links the already-built result instead of running the slow build script again.

Every file in the pnpm store is verified using a checksum. This ensures that the code on your disk exactly matches what was published to npm, protecting you from "disk corruption" or "supply chain" tampering where a local file might have been modified.

Thank you,

X - https://x.com/06_Shreehari

LinkedIn - https://www.linkedin.com/in/shreehari-acharya/

GitHub - https://github.com/Shreehari-Acharya

LLMs are great at answering questions because they have been trained on a lot of data. But how can we train them on our personal data? Personal data could be an organization's internal knowledge base or all the articles and blogs you have written. Injecting this personal data into Large Language Models (LLMs) and then getting answers back is what we can call RAG in the simplest terms.

Why RAG ?

Can't we just paste our personal data into the prompt if we need answers about our data?

Yes, you could, and that's an efficient method when your data is small and fits within the context window. (The context window refers to the maximum size of a prompt that can be sent.)

However, PDFs are often around 50-100 pages with lots of text. Can you paste all of that into the prompt? That would be very inefficient.

But I always got an answer when I uploaded a really big document. How did that happen?

Yes, uploading a document works because it uses RAG behind the scenes. However, pasting the contents directly into the prompt would be inefficient.

How does RAG work ?

Lets now understand how RAG works step by step

Step 1 - Chunking

Chunking refers to the process of splitting large amounts of data into smaller portions

How to chunk a document is a complex process. There isn't a universal size for a chunk. It's up to the developers to experiment with different methods to determine how they should chunk the data.

Step 2 - Indexing

Indexing is the process of storing this chunks in a way that allows us to retrieve them efficiently

Suppose you wanted to know if a big PDF contained a poem written by Shakespeare. Let's assume that the poem and author details are stored in a specific chunk. If we could retrieve just that chunk, we could easily answer your query by providing the data from that chunk to the LLM. Of course, it would then confirm it.

Indexing achieves this by first converting all the chunks into vector embeddings and then storing them in a vector database.

Vector embeddings turn things like text or images into numbers, so that similar things are close together on a graph. For example, the words king and queen will be close in this space because they have similar meanings. This helps computers understand relationships between words, images, or other data.

How do we convert them into vector embeddings?

Many AI companies that own an LLM have their own vector embeddings. We can use any of these to convert the data into vector embeddings. Some are proprietary, like OpenAI, while others are open source, such as those from Hugging Face.

Step 3 - Storing

We now need to store these sets of embeddings in a database. It's important to note that we can't just use our regular No-SQL or SQL databases. We need a different type of database that is efficient at storing these embeddings, allowing us to perform similarity searches and other operations.

A database that stores vectors and allows us to perform related operations is called a Vector Database

Here is a list of popular Vector databases

• Pinecone

• Qdrant

• pgvector

• Milvas

Step 4 - Embedding of User query

A user always has a question or wants information from this document. To find the relevant parts, we need to perform a similarity search. Therefore, we first need to convert the user's query into embeddings.

Note: We will need to use the same embedding model

Step 5 - Retrieval of relevant chunks

Once the user's query is converted into embeddings, we perform a similarity search using these embeddings. This returns embeddings along with data that contains similar information.

Step 5 - Passing the retrieved data and the original user query to LLM.

Now we have the relevant chunk, which hopefully contains a poem by Shakespeare and his name. We will send this data as context to the LLM, along with the original user query. The LLM should then respond by confirming and displaying the poem.

And that’s how a simple Retrieval-Augmented Generation (RAG) system operates behind the scenes. By now, you should have a solid understanding of the fundamental components and processes involved in constructing a basic RAG-based application. This includes everything from embedding user queries to performing similarity searches and leveraging vector databases to retrieve relevant information. In my next article, we will dive deeper into this topic and work together to build a practical RAG application step by step. This hands-on approach will help solidify your understanding and give you the confidence to implement RAG systems in real-world scenarios.

Thankyou,

My Socials

• X - https://x.com/06_Shreehari

• LinkedIn - https://www.linkedin.com/in/shreehari-acharya/

• GitHub - https://github.com/Shreehari-Acharya

Model : Its referring to the Large Language Model (LLM)

Context : Explaining what and how, which is required for the model to produce great answers

Protocol : Just a set of rule on how to do something.

So MCP is a set of rules that tells the Large language model what and how to use certain tools.

Understanding a Tool

When we write a program, we create functions, and then need to call the function explicitly to run it.

from datetime import date

def get_current_date(): # A simple function that returns the current date
    return date.today()

if __name__ == "__main__":
    print("Today's date is:", get_current_date()) # calling the function

What happens, when we ask the current date/time to chatGPT?

See.. It could not give us the current time because it does not have access to real-time data. But our small program does have a function which can give me the date.

What if we could somehow give chatGPT the ability to run our custom function?

That is what a Tool is! Its just another name for a particular function

MCP Client & MCP Server

Let us now understand the difference between MCP Client and MCP Server

MCP Client

The browser knows how to interact with a web-server. It knows the HTTP and thus can talk to any web server. Similarly a MCP Client knows how to talk to a MCP Server. Some of the popular applications have MCP Client. Cluade desktop, Cursor, Windsurf are among the popular ones.

MCP Server

The MCP server contains these Tools which LLM can call to execute certain actions. It also describes the tool purpose, how it is used, what parameters does it take, and what does it return on successful execution.

Final flow

Let's suppose we have an MCP Server with just one tool called current-date, which calls our defined function get_current_date().

The first step is to communicate with the MCP Server to get all the tools, their description, the parameters they take, and what they return upon successful execution. We do this using the MCP Client. The next step is to take the user's query and pass it along with the list of tools and their descriptions. Now, the LLM can call the tools if it needs specific information that a tool can provide, using the MCP Client to access real-time data!

Diagram explaining the entire process

Step 1

step 2

Step 3

Step 4

Step 5

Final step: ChatGPT gives back the final output to the user

Where can it be useful?

It can be used in a lot of places. You could have a MCP-server hosting your tools, which may be functions to post something on social media, or send an email, or whatever you can imagine. Then just ask the AI which has access to your MCP server to do some task, and if needed it will call the tool and can even post a tweet on behalf of you!.

Larger use cases

Many companies have already started building their own MCP servers, so that their users can interact with their products directly in human language. Here is a list of such amazing servers, that can perform some amazing tasks. MCP-servers-list

Further materials

Official documentation - https://modelcontextprotocol.io/introduction

creating a MCP server to buy/sell stocks - https://youtu.be/1iJ34tTjwwo?si=7asblcnGFztWK7ou

Thank you,

X - https://x.com/06_Shreehari

LinkedIn - https://www.linkedin.com/in/shreehari-acharya/

GitHub - https://github.com/Shreehari-Acharya