Awesome
Generative AI Document Summarization
Description
Tagline
Create summaries of a large corpus of documents using Generative AI.
Detailed
This solution showcases how to summarize a large corpus of documents using Generative AI. It provides an end-to-end demonstration of document summarization going all the way from raw documents, detecting text in the documents and summarizing the documents on-demand using Vertex AI LLM APIs, Cloud Vision Optical Character Recognition (OCR) and BigQuery.
PreDeploy
To deploy this blueprint you must have an active billing account and billing permissions.
Architecture
- The developer follows a tutorial on a Jupyter Notebook, where they upload a PDF — either through Vertex AI Workbench or Colaboratory.
- The uploaded PDF file is sent to a function running on Cloud Functions. This function handles PDF file processing.
- The Cloud Functions function uses Cloud Vision to extract all text from the PDF file.
- The Cloud Functions function stores the extracted text inside a Cloud Storage bucket.
- The Cloud Functions function uses Vertex AI’s LLM API to summarize the extracted text.
- The Cloud Functions function stores the text summaries of PDFs in BigQuery tables.
- As an alternative to uploading PDF files through Jupyter Notebook, the developer can upload a PDF file directly to a Cloud Storage bucket — for instance, through the Console UI or gcloud. This upload triggers Eventarc to begin the Document Processing phase.
- As a result of the direct upload to Cloud Storage, Eventarc triggers the Document Processing phase, handled by Cloud Functions.
Documentation
Deployment Duration
Configuration: 1 mins Deployment: 10 mins
Cost
<!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->Inputs
| Name | Description | Type | Default | Required |
|---|---|---|---|---|
| bucket_name | The name of the bucket to create | string | "genai-webhook" | no |
| gcf_timeout_seconds | GCF execution timeout | number | 900 | no |
| project_id | The Google Cloud project ID to deploy to | string | n/a | yes |
| region | Google Cloud region | string | "us-central1" | no |
| time_to_enable_apis | Wait time to enable APIs in new projects | string | "180s" | no |
| webhook_name | Name of the webhook | string | "webhook" | no |
| webhook_path | Path to the webhook directory | string | "webhook" | no |
Outputs
| Name | Description |
|---|---|
| genai_doc_summary_colab_url | The URL to launch the notebook tutorial for the Generateive AI Document Summarization Solution |
| neos_walkthrough_url | The URL to launch the in-console tutorial for the Generative AI Document Summarization solution |
Requirements
These sections describe requirements for using this module.
Software
The following dependencies must be available:
- Terraform v0.13
- Terraform Provider for GCP plugin v3.0
Service Account
A service account with the following roles must be used to provision the resources of this module:
- Storage Admin:
roles/storage.admin
APIs
A project with the following APIs enabled must be used to host the resources of this module:
- Google Cloud Storage JSON API:
storage-api.googleapis.com
Contributing
Refer to the contribution guidelines for information on contributing to this module.
Security Disclosures
Please see our security disclosure process.