Skip to content

Latest commit

 

History

History
143 lines (108 loc) · 5.32 KB

data.md

File metadata and controls

143 lines (108 loc) · 5.32 KB

BACK Translation of Func

Answer:

def update_torch_dtype(self, torch_dtype):
    if torch_dtype is None:
        torch_dtype = torch.float16
        logger.info("Loading the model in `torch.float16`. To overwrite it, set `torch_dtype` manually.")
    elif torch_dtype == torch.bfloat16 and (torch.cuda.is_available() or torch.xpu.is_available()):
        logger.warning(
            "`torch.bfloat16` is not supported for AWQ CUDA/XPU kernels yet. Casting to `torch.float16`."
        )
        torch_dtype = torch.float16
    elif torch_dtype != torch.float16 and (torch.cuda.is_available() or torch.xpu.is_available()):
        logger.warning(
            "We suggest you to set `torch_dtype=torch.float16` for better efficiency on CUDA/XPU with AWQ."
        )
    return torch_dtype

AWQ Quantization Code Completion Challenge

You are integrating a 4-bit quantized model using HuggingFace's transformers and the autoawq library. While inspecting the AwqQuantizer class defined in the quantization module (e.g., quantization/awq.py), you notice a method named update_torch_dtype is missing.

Your task is to implement the update_torch_dtype method of AwqQuantizer, based on the following behavior requirements:

  1. The method should receive a torch_dtype as input and return the updated torch_dtype.
  2. If no dtype is provided (None), it should default to torch.float16 and emit an info log.
  3. If torch_dtype is torch.bfloat16 and the device is CUDA or XPU, log a warning and downgrade to torch.float16, because bfloat16 is not yet supported.
  4. If a non-float16 dtype is provided and CUDA/XPU is available, it should warn that float16 is recommended for best performance.
  5. You may assume torch and logger are already available in scope, and is_torch_available() has been checked.
  6. This logic must be compatible with multiple backends (CPU, CUDA, XPU), so proper device checks are necessary.

Hints:

  • You may explore how similar dtype handling is done across other quantization backends in the HuggingFace quantization module.
  • Device availability checks may look like torch.cuda.is_available() or torch.xpu.is_available().

Fill in the missing method.

Comments: Can only do sft, llm-as-judge reward, or we let model write unit test.

Back Translation of Examples/ Tutorial

https://github.com/huggingface/transformers/blob/main/examples/quantization/custom_quantization_int8_example.py

name: int8_symmetric_quantization description: > Implements a custom symmetric INT8 quantization workflow using Hugging Face Transformers. The task includes defining a quantized linear layer, recursively replacing Linear layers, registering a custom quantization config and quantizer, and applying the quantizer to a Llama model. tags:

  • transformers
  • quantization
  • int8
  • model-compression
  • llama
  • huggingface

components: quantized_linear_layer: class_name: Int8SymmetricLinear description: > Custom linear layer that uses symmetric INT8 quantization with per-row scaling. parameters: - in_features - out_features - bias - dtype

quantization_config: class_name: Int8SymmetricConfig description: > Defines which modules to exclude and specifies the quantization method as 'int8_symmetric'.

quantizer: class_name: Int8SymmetricQuantizer description: > Implements the logic to replace standard linear layers with quantized ones, quantize weights, and integrate with the Transformers framework.

layer_replacement_function: name: replace_with_int8_symmetric_linear description: > Recursively replaces nn.Linear layers in the model with Int8SymmetricLinear, skipping specified layers (e.g., lm_head).

workflow:

  • load_model: model_id: meta-llama/Llama-3.2-1B quantization_config: Int8SymmetricConfig dtype: float32 device_map: cpu

  • inference: input_text: "once there is" tokenizer: AutoTokenizer generation_config: max_length: 100 num_return_sequences: 1 no_repeat_ngram_size: 2

  • save_model: output_dir: Llama-3.2-1B-INT8-CUSTOM

  • upload_to_hf_hub: repo_id: medmekk/Llama-3.2-1B-INT8-CUSTOM private: false

output: model_path: Llama-3.2-1B-INT8-CUSTOM repo_url: https://huggingface.co/medmekk/Llama-3.2-1B-INT8-CUSTOM

requirements:

  • torch
  • accelerate
  • transformers >= 4.38.0
  • huggingface_hub

Comments: example code is very limited, and the model can read pydoc to hack.

Deep Research For Func by Back Translation

Suppose you're building a vision-language model with GLM in repo transformers that expects a fixed number of visual tokens from a video input. You want to systematically extract frames, resize and normalize them, divide them into space-time patches, and format them into model-ready tensors — all while keeping track of the temporal locations. What kind of preprocessing module or strategy would you use for that?

Answer: Glm4vVideoProcessor._preprocess

Comments: Clear reward, but the objective is restricted, only can train for information seeking.

Deep Research For Func by DeepWiki

Question: Where is the MOE Router for Qwen Answer: src/transformers/models/qwen2_moe/modeling_qwen2_moe.py; 582-596

Comments: Clear reward, but deepwiki may restrict the RPM.