Meeting 1

(May 30,2024)

Attendees:

• Kaushlendra Pratap
• Anupam Ghosh
• Gaurav Mishra
• Shaheem Azmal M MD
• Ayush Bhardwaj
• Avinal Kumar

Discussion:

LLM Comparison Codebase

I presented a preliminary codebase designed to interface with multiple LLM APIs (Groq, Nvidia, Together AI) for comparison purposes. This allows us to assess the performance of different models on our license identification task.

Initial PyTorch Dataset

I prepared a small PyTorch dataset using the Nomos agent to evaluate the LLMs.

Codebase Structure

The project repository is structured as follows: GSOC24

GSOC24
├── helpers
│   ├── functions
│   ├── llm_client
│   ├── models
│   └── parsers

functions: Contains helper functions for sampling, file operations, etc.

llm_client: Central class for interacting with LLMs, including API initialization, logging, and retry mechanisms. This week focused on laying the groundwork for efficient, automated requests to the APIs.

models: Enum class for supported LLM models (LLama 8b, Mistral 7b, Gemma 1 7b, Phi_3 mini/small).

parsers: (Placeholder for future LLM output parsing functions)

Next Steps

Refined Data Input

Instead of feeding the entire file comments to the LLM, we will use Nirjas to extract and provide only the comments relevant to license information. This should improve accuracy and reduce computational costs. Enhanced Dataset: We plan to create a more diverse dataset using Linux code, as this will better represent the variety of license structures and wording encountered in real-world projects.

Codebase Enhancements

Finalize the llm_client functionality to enable efficient, automated requests across the entire dataset.