vllm.models.common.ops.sequence_parallel

Functions:

• is_sequence_parallel_active –

  Whether a batch of num_tokens shards the stream along the sequence.

is_sequence_parallel_active(is_enabled, num_tokens, min_tokens=0) cached

Whether a batch of num_tokens shards the stream along the sequence.

Below one token per rank the shards are mostly padding, so the batch stays replicated. A model may raise this lower bound with min_tokens.

Source code in vllm/models/common/ops/sequence_parallel.py

@lru_cache
def is_sequence_parallel_active(
    is_enabled: bool, num_tokens: int, min_tokens: int = 0
) -> bool:
    """Whether a batch of ``num_tokens`` shards the stream along the sequence.

    Below one token per rank the shards are mostly padding, so the batch
    stays replicated. A model may raise this lower bound with ``min_tokens``.
    """
    return is_enabled and num_tokens >= max(
        get_tensor_model_parallel_world_size(), min_tokens
    )