GRASS: Gradient-based Adaptive Layer-wise Importance Sampling for Memory-efficient Large Language Model Fine-tuning

Implement GRASS for memory-efficient fine-tuning. This provides an alternative to LoRA that maintains more expressiveness, which is highly sought after by small teams.

Suggested repo: grass-train

"Get full-parameter performance with fractional memory using gradient-based sampling."

Estimated effort: 100h