Research notes

Personal summaries and insights gathered from reading various research papers and articles.

Training Data Influence Analysis and Estimation A Survey

Machine Learning
📝 Paper: https://arxiv.org/pdf/2212.04612.pdf

5 min

Self-Rewarding Language Models

Large Language Models
This paper introduces Iterative DPO and use it to improve the performance of Llama 2 70B on the AlpacaEval…

4 min

Local Large Language Models – Int8

Large Language Models
Quantization
📝 Article: https://int8.io/local-large-language-models-beginners-guide/

6 min

LoraHub – Efficient Cross-Task Generalization via Dynamic LoRA Composition

Large Language Models
This paper describes a framework to combine LoRA modules to achieve adaptable performance on unseen tasks.…

2 min

Multipack Sampler

Large Language Models
💻 GitHub: https://github.com/imoneoi/multipack_sampler

2 min

InCoder – A Generative Model for Code Infilling and Synthesis

Large Language Models
InCoder is a 1.3B parameter LLM that can generate code (left-to-right) as well as editing (via masking and…

3 min

Orca – Progressive Learning from Complex Explanation Traces of GPT-4

Large Language Models
Orca is a 13B parameter LLM with ChatGPT level of performance thanks to a huge dataset of 5M samples with…

6 min

phi-1 – Textbooks Are All You Need

Large Language Models
It introduces phi-1, a model with 1.3B parameters that obtains a pass@1 rate of 50.6% on HumanEval thanks to…

4 min

LongNet – Scaling Transformers to 1,000,000,000 Tokens

Large Language Models
This paper introduces the dilated attention mechanism, another sparse attention scheme which approximates…

3 min

Tart – A plug-and-play Transformer module for task-agnostic reasoning

Large Language Models
Tart combines the performance of fine-tuning with the ease-of-use of in-context learning. It is a general…

4 min

Inference Optimization – Lil’Log

Large Language Models
Quantization
📝 Article: https://lilianweng.github.io/posts/2023-01-10-inference-optimization/

5 min

Extending the Context Window of LLMs

Large Language Models
📝 Article: https://kaiokendev.github.io/context

2 min

GPTQ – Accurate Post-Training Quantization for Generative Pre-trained Transformers

Large Language Models
This paper introduces GPTQ, the first reliable quantization technique to peform 4- and even 3-bit…

7 min

LIMA – Less Is More for Alignment

Large Language Models
LIMA is a 65B parameter LLaMA model fine-tuned (supervised learning) using 1,000 samples, which outperforms…

4 min

LoRA – Low-Rank Adaptation of Large Language Models

Large Language Models
PEFT approach based on matrix factorization.

3 min

Report – Few-Shot Text Classification

Large Language Models
📝 Article: https://few-shot-text-classification.fastforwardlabs.com/ (2020)

2 min
No matching items