text-generation

library0.7.0pypypi✓ verified 89d ago

The `text-generation` library is the official Python client for interacting with the Hugging Face Text Generation Inference (TGI) backend, a highly optimized solution for deploying large language models. It provides synchronous and asynchronous APIs for text generation, including streaming capabilities. The current version is 0.7.0, and while the underlying TGI server has a rapid release cadence with frequent updates, the client library itself is updated less often, focusing on stability and compatibility with common TGI server versions.

llm-agents ai-ml http-networking

pip install text-generation

github ↗package ↗

INSTALL

IMPORT

SIG · TEXT-GENERATION

text-generation

llm-agentspythonv0.7.0

Install

7.4s avg

Import

1610ms

Disk

62MB

Pass rate

10/ 10

Env Coverage10 / 10

glibc

3.9–3.13

musl

3.9–3.13

Install & Compatibility

Where this runs

tested against v0.7.0 · pip install

no network on importno background threads

Install × environment matrix

Each cell = how many times install + import succeeded across repeated harness runs. Partial = flaky.
glibc = Debian/Ubuntu slim · musl = Alpine Linux

musl

py 3.10–3.910 runs

installs and imports cleanly · install 0.0s · import 1.702s · 61.5MB

glibc

py 3.10–3.910 runs

installs and imports cleanly · install 7.4s · import 1.517s · 63MB

62MB installed

● package 62MB

Code

Verified usage

Verified import paths — ran on the pinned version, not inferred.

Client

✓ from text_generation import Client

AsyncClient

✓ from text_generation import AsyncClient

InferenceAPIError

✓ from text_generation import InferenceAPIError

✗ from text_generation.errors import InferenceAPIError

The error class was directly exposed under the top-level package in recent versions for simpler import.

This quickstart demonstrates how to instantiate a `Client` for synchronous text generation. It includes examples for both single-response generation and streaming token-by-token. Remember to set the `TGI_ENDPOINT` environment variable or directly provide the correct URL for your running Text Generation Inference server.

import os from text_generation import Client, InferenceAPIError # Ensure the TGI server is running and accessible at this URL # For example, a local server might be 'http://127.0.0.1:8080' # or a deployed endpoint 'https://your-tgi-endpoint.huggingface.cloud' TGI_ENDPOINT = os.environ.get('TGI_ENDPOINT', 'http://127.0.0.1:8080') try: client = Client(TGI_ENDPOINT) # Generate a single response response = client.generate( "What is the capital of France?", max_new_tokens=20, repetition_penalty=1.05 ) print(f"Generated Text: {response.generated_text}") print("\n--- Streaming Example ---") # Stream tokens for response in client.generate_stream( "Write a short poem about a cat.", max_new_tokens=50 ): if not response.token.special: print(response.token.text, end="", flush=True) print("\n") except InferenceAPIError as e: print(f"Error from TGI server: {e}") except Exception as e: print(f"An unexpected error occurred: {e}")

Debug

Known issues

breakingThe `text-generation` client is tightly coupled with the `text-generation-inference` (TGI) server. Breaking changes in the TGI server's API (e.g., changes to request/response schemas or new required parameters) may necessitate an update to the client library, even if the client's own version hasn't changed drastically.

fix

Always ensure your `text-generation` client version is compatible with your deployed `text-generation-inference` server version. Refer to the TGI server's documentation for API stability notes and recommended client versions.

affects: All versions, depends on TGI server version

gotchaIncorrect or inaccessible `base_url` for the `Client` will lead to connection errors. The default `http://127.0.0.1:8080` is only for a local server running with default settings.

fix

Verify that the `base_url` provided to `Client()` or `AsyncClient()` points to a running and network-accessible TGI endpoint. Double-check port numbers, hostnames, and any necessary authentication if applicable.

affects: All versions

gotchaDifferentiating between `client.generate()` and `client.generate_stream()` is crucial. `generate()` returns a single `Response` object with the full generated text after completion. `generate_stream()` returns an iterable that yields `StreamResponse` objects as tokens are generated.

fix

Use `generate()` for single, complete outputs where latency isn't a critical concern for the first token, or for batching. Use `generate_stream()` for real-time applications where displaying tokens as they are produced enhances user experience.

affects: All versions

gotchaServer-side errors are encapsulated in `InferenceAPIError`. These errors often contain specific details from the TGI server, such as issues with model loading, invalid parameters, or GPU memory exhaustion.

fix

Always wrap your client calls in a `try...except InferenceAPIError` block. Log the error message to understand the root cause of the server-side failure. Check your request parameters against the model's capabilities and server configuration.

affects: All versions

Errors

Common errors & fixes

httpx.ConnectError: [Errno 111] Connection refused

The Python client could not establish a connection to the Text Generation Inference server at the specified `base_url`. This typically means the server is not running, is running on a different port/host, or a firewall is blocking the connection.

fix

Ensure your TGI server is running and accessible from where you are running the client. Verify the `base_url` (e.g., `http://127.0.0.1:8080`) matches the server's actual address and port.

text_generation.InferenceAPIError: {'error': '...', 'error_type': '...'}

This error indicates that the TGI server received your request but encountered an issue processing it, returning an HTTP error status (e.g., 400, 404, 500). The `error` and `error_type` fields in the message usually provide specifics.

fix

Examine the error message within the `InferenceAPIError`. Common causes include: invalid model ID, insufficient GPU memory, unsupported generation parameters, or issues during model loading. Check server logs for more detailed diagnostics.

TypeError: Client.__init__() missing 1 required positional argument: 'base_url'

You are attempting to instantiate the `Client` class without providing the `base_url` argument, which is mandatory.

fix

Always provide the URL of your Text Generation Inference server when creating a `Client` instance, e.g., `client = Client('http://your-tgi-server:8080')`.

AttributeError: module 'text_generation' has no attribute 'Client'

This usually indicates an outdated `text-generation` library installation, a conflicting package name in your Python environment, or an incorrect import path.

fix

Ensure you have the latest `text-generation` library installed (`pip install --upgrade text-generation`). Verify your import statement is `from text_generation import Client` and check for any local files named `text_generation.py` that might be shadowing the installed library.

Upgrade

Version history

0.7.0latest on PyPI · released Mar 22, 2024

Audit

Dependencies

httpxrequiredUsed for making synchronous and asynchronous HTTP requests to the TGI server.

pydanticrequiredUsed for data validation and parsing of request and response models.

Agent activity

26 hits · last 30 days

node

OpenAI (training)

Resources

githubhttps://github.com/huggingface/text-generation-inference ↗

packagehttps://pypi.org/project/text-generation/ ↗