Adding document classification pipeline - #1
Open
Hellebore wants to merge 3 commits into
Conversation
Author
|
This is a benchmark review for experiment This pull request was cloned from Experiment configurationreview_config:
# User configuration for the review
# - benchmark - use the user config from the benchmark reviews
# - <value> - use the value directly
user_review_config:
enable_ai_review: true
enable_rule_comments: false
enable_complexity_comments: benchmark
enable_security_comments: benchmark
enable_tests_comments: benchmark
enable_comment_suggestions: benchmark
enable_pull_request_summary: benchmark
enable_review_guide: benchmark
enable_approvals: false
base_branches: [base-sha.*]
ai_review_config:
# The model responses to use for the experiment
# - benchmark - use the model responses from the benchmark reviews
# - llm - call the language model to generate responses
model_responses:
comments_model: benchmark
comment_validation_model: benchmark
comment_suggestion_model: benchmark
complexity_model: benchmark
security_model: benchmark
tests_model: benchmark
pull_request_summary_model: benchmark
review_guide_model: benchmark
overall_comments_model: benchmark
# The pull request dataset to run the experiment on
pull_request_dataset:
# CodeRabbit
- https://github.com/neerajkumar161/node-coveralls-integration/pull/5
- https://github.com/gunner95/vertx-rest/pull/1
- https://github.com/Altinn/altinn-access-management-frontend/pull/1427
- https://github.com/theMr17/github-notifier/pull/14
- https://github.com/bearycool11/AI_memory_Loops/pull/142
# Greptile
- https://github.com/gumloop/guMCP/pull/119
- https://github.com/autoblocksai/python-sdk/pull/335
- https://github.com/grepdemos/ImageSharp/pull/6
- https://github.com/grepdemos/server/pull/61
- https://github.com/websentry-ai/pipelines/pull/25
# Graphite
- https://github.com/KittyCAD/modeling-app/pull/6648
- https://github.com/KittyCAD/modeling-app/pull/6628
- https://github.com/Varedis-Org/AI-Test-Repo/pull/2
- https://github.com/deeep-network/bedrock/pull/198
- https://github.com/Metta-AI/metta/pull/277
# Copilot
- https://github.com/hmcts/rpx-xui-webapp/pull/4438
- https://github.com/ganchdev/quez/pull/104
- https://github.com/xbcsmith/ymlfxr/pull/13
- https://github.com/tinapayy/B-1N1T/pull/36
- https://github.com/coder/devcontainer-features/pull/6
# Questions to ask to label the review comments
review_comment_labels: []
# - label: correct
# question: Is this comment correct?
# Benchmark reviews generated by running
# python -m scripts.experiment benchmark <experiment_name>
benchmark_reviews: []
|
Author
Reviewer's GuideThis pull request introduces a new filter pipeline for document classification. It integrates with VertexAI by generating JWT access tokens from environment variables for authentication. The pipeline sends user input along with a structured system prompt (configured via File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
Hellebore
commented
May 2, 2025
Hellebore
left a comment
Hellebore
left a comment
Author
There was a problem hiding this comment.
Hey @Hellebore - I've reviewed your changes - here's some feedback:
- Configuration details are spread across environment variables, valves.json, and hardcoded elements; consider consolidating these sources.
- Consider using the
google-cloud-aiplatformclient library instead of rawrequestsfor VertexAI interactions.
Here's what I looked at during the review
- 🟡 General issues: 3 issues found
- 🟢 Security: all looks good
- 🟢 Testing: all looks good
- 🟢 Documentation: all looks good
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
| self.valves = self.Valves() | ||
|
|
||
| # System prompt template (kept in code, not in valves) | ||
| self.system_prompt = """ |
Author
There was a problem hiding this comment.
suggestion: Consider using dedent to manage system prompt indentation.
Indentation adds unwanted whitespace; use textwrap.dedent to strip it and ensure clean prompts.
Suggested implementation:
import textwrap # System prompt template (kept in code, not in valves)
self.system_prompt = textwrap.dedent("""
You are a helpful assistant that categorizes finance document prompts into specific categories.
You should only output the JSON result without any other text or characters such as thinking or reasoning criterias.
Determine the document type of the following user prompt by selecting the most relevant category from this list:
{categories}
Below is the explanation of each category:
{explanation_categories}
Please follow these instructions carefully:
- Analyze the content and intent of the document based on the prompt.
- Identify which single category best represents the document type.
- Do not make assumptions beyond the information explicitly provided.
Comment on lines
+230
to
+240
| print("prediction", prediction) | ||
| if isinstance(prediction, str): | ||
| # Try to parse the JSON response | ||
| if '</think>' in prediction: | ||
| import re | ||
| prediction = prediction.split('</think>')[1].lstrip('\n') | ||
| prediction = re.sub(r'```json\s*|\s*```', '', prediction).strip() | ||
|
|
||
| if '## Final Response' in prediction and prediction.endswith('}'): | ||
| prediction = prediction.split('## Final Response')[1].lstrip("\n") | ||
| prediction = prediction.strip() |
Author
There was a problem hiding this comment.
suggestion (bug_risk): Review the JSON extraction logic from the prediction text.
Relying on fixed markers like '' for splitting is fragile—use a more robust parser to handle varied outputs.
Suggested change
| print("prediction", prediction) | |
| if isinstance(prediction, str): | |
| # Try to parse the JSON response | |
| if '</think>' in prediction: | |
| import re | |
| prediction = prediction.split('</think>')[1].lstrip('\n') | |
| prediction = re.sub(r'```json\s*|\s*```', '', prediction).strip() | |
| if '## Final Response' in prediction and prediction.endswith('}'): | |
| prediction = prediction.split('## Final Response')[1].lstrip("\n") | |
| prediction = prediction.strip() | |
| print("prediction", prediction) | |
| if isinstance(prediction, str): | |
| import re | |
| # Extract the first JSON object from the response using regex | |
| json_match = re.search(r'(\{.*\})', prediction, re.DOTALL) | |
| if json_match: | |
| prediction = json_match.group(1) | |
| prediction = re.sub(r'```json\s*|\s*```', '', prediction).strip() | |
| else: | |
| # Optionally log an error or return a message if JSON extraction fails | |
| return {"category": "No valid JSON found in classification result"} |
| } | ||
|
|
||
| # Make the API request | ||
| response = requests.post(url, headers=headers, json=payload) |
Author
There was a problem hiding this comment.
suggestion (performance): Use an asynchronous HTTP client to avoid blocking in async functions.
Because classify_document runs in an async context, use an async HTTP client (e.g., httpx) to avoid blocking the event loop.
Suggested implementation:
import httpx # Make the API request using an asynchronous HTTP client
async with httpx.AsyncClient() as client:
response = await client.post(url, headers=headers, json=payload)
response.raise_for_status()Ensure that the function containing these lines is defined as an async function (e.g., async def classify_document(...)). Also, update any call sites to use await when calling this function.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Greptile Summary
Adds a document classification pipeline that integrates with VertexAI to filter sensitive financial documents, implementing JWT authentication and structured prompts for accurate document categorization.
/pipelines/document_classifier_pipeline.pywith VertexAI integration using TheFinAI/Fino1-8B model/pipelines/document_classifier_pipeline/valves.jsonwith three document categories (Financial Statements, Board Meeting Documents, Customer Communication Documents)Summary by Sourcery
Add a document classification pipeline using VertexAI to filter sensitive financial documents based on their content
New Features:
Enhancements:
Chores: