Enhance Token Count Estimation in AI Response Generation

[jkpe]

This pull request introduces a new utility function to estimate the token count of prompts and responses in the AI response generation process. This enhancement aims to provide better insights into the token usage, which is crucial for optimizing performance and managing costs associated with API calls.

Changes Made

• Added _estimate_token_count Function:

  • This function provides a rough estimate of the token count based on the number of words in the input text. The approximation used is that 1 token is roughly equivalent to 0.75 words.

• Updated get_provider_response Function:

  • Integrated the token count estimation into the response generation process.
  • Added logging to display the estimated token count for both the context and the generated response, enhancing transparency and debugging capabilities.

Benefits

• Improved Monitoring: By estimating token counts, developers can better understand the resource usage of the AI model, allowing for more informed decisions regarding prompt design and context management.
• Enhanced Debugging: The additional logging provides clearer insights into the response generation process, making it easier to identify potential issues related to token limits.

Notes

• The token estimation is a rough approximation and may not reflect the exact token count used by the underlying model. Further refinements can be made in future iterations if needed.

---