Infrastructure¶
You can provision infrastructure within your Goblet code.
Redis¶
app = Goblet()
app.redis("redis-example")
When deploying a backend, the environment variables will automatically be updated to include the REDIS_INSTANCE_NAME, REDIS_HOST and REDIS_PORT. To further configure your Redis Instance within Goblet, specify the `redis` key in your config.json. You can reference Redis Instance Resource for more information on available fields.
VPC Connector¶
app = Goblet()
app.vpcconnector("vpcconnector")
When deploying a backend, the vpc access configuration will be updated to include the specified vpc connector. To further configure your VPC Connector within Goblet, specify the `vpcconnector` key in your config.json. You can reference Connector Resource for more information on available fields.
Note
In order to ensure proper configuration of the VPC Connector, the ipCidrRange key is required to be set within vpcconnector of your config.json.
Api Gateway¶
app = Goblet(function_name="openapi-existing")
filename = "openapi_spec.yml"
app.apigateway("openapi-existing", "BACKEND_URL", filename=filename)
app = Goblet(function_name="openapi-existing-dict")
openapi_dict = {
"swagger": "2.0",
"info": {
"title": "media-serving-service",
"description": "Goblet Autogenerated Spec",
"version": "1.0.0",
},
"schemes": ["https"],
"produces": ["application/json"],
"paths": {
"/": {
"get": {
"operationId": "get_main",
"responses": {"200": {"description": "A successful response"}},
}
}
},
"definitions": {},
}
app.apigateway("openapi-existing-from-dict", "BACKEND_URL", openapi_dict=openapi_dict)
You can deploy an Api Gateway and all related resources (Api, Api Config, Gateway) with an existing openapi spec using the apigateway decorator. The decorator can take in a filename or the spec as a dictionary. If you dont need to deploy any other Goblet resources you can deploy just the Api Gateway using goblet deploy -p PROJECT –skip-backend –skip-resources
By default there is a timeout on Api Gateway of 15 seconds. This can be overriden by setting “api_gateway”: {“deadline”: 45} in config.json.
Note
API Gateway only supports swagger 2.0.
CloudTask Queue¶
app = Goblet()
config = { ... }
client = app.cloudtaskqueue("queue", config=config)
To further configure your CloudTaskQueue within Goblet, provide the config parameter base on the documentation. CloudTask Queue Resource. The configuration can be provided inline when declaring the queue, or in your config.json under the cloudtaskqueue key.
{
"cloudtaskqueue": {
"queue": { ... }
}
}
PubSub Topics¶
app = Goblet()
config = { ... }
app.pubsub_topic("topic", config=config)
To further configure your PubSub topic within Goblet, provide the config parameter base on the documentation. Topic Resource.
BigQuery Spark Stored Procedures¶
To deploy BigQuery stored procedures using Spark follow the example below. BigQuery stored procedures documentation can be found here.
Using a function from the same python file: .. code:: python
import logging from goblet import Goblet, goblet_entrypoint import pyspark.sql.functions as F from pyspark.sql import SparkSession
app = Goblet(function_name=”create-bq-spark-stored-procedure”)
app.log.setLevel(logging.DEBUG) # configure goblet logger level goblet_entrypoint(app)
# Create a bq spark stored procedure with the spark code and additional python files def spark_handler():
spark = SparkSession.builder.appName(“spark-bigquery-demo”).getOrCreate()
# Load data from BigQuery. texts = spark.read.format(“bigquery”) .option(“table”, “tutorial.poc”) .load() texts.createOrReplaceTempView(“words”)
# Perform word count. text_count = texts.select(“id”, “text”, F.length(“text”).alias(“sum_text_count”)) text_count.show() text_count.printSchema()
# Saving the data to BigQuery text_count.write.mode(“append”).format(“bigquery”) .option(“writeMethod”, “direct”) .save(“tutorial.wordcount_output”)
- app.bqsparkstoredprocedure(
name=”count_words_procedure_external”, dataset_id=”tutorial”, func=spark_handler)
Using a function from a different python file and loading additional python files: spark.py: .. code:: python
- def spark_handler():
from pyspark.sql import SparkSession import pyspark.sql.functions as F spark = SparkSession.builder.appName(“spark-bigquery-demo”).getOrCreate()
# Load data from BigQuery. texts = spark.read.format(“bigquery”) .option(“table”, “tutorial.poc”) .load() texts.createOrReplaceTempView(“words”)
# Perform word count. text_count = texts.select(“id”, “text”, F.length(“text”).alias(“sum_text_count”)) text_count.show() text_count.printSchema()
# Saving the data to BigQuery text_count.write.mode(“append”).format(“bigquery”) .option(“writeMethod”, “direct”) .save(“tutorial.wordcount_output”)
- if __name__ == “__main__”:
spark_handler()
additional.py: .. code:: python
import logging logging.basicConfig(level=logging.INFO) logger = logging.getLogger(__name__)
- def additional_func():
logger.info(“additional_func”)
main.py: .. code:: python
import logging from goblet import Goblet, goblet_entrypoint
app = Goblet(function_name=”create-bq-spark-stored-procedure”)
app.log.setLevel(logging.DEBUG) # configure goblet logger level goblet_entrypoint(app)
# Create a bq spark stored procedure with the spark code and additional python files app.bqsparkstoredprocedure(name=”count_words_procedure_external”, dataset_id=”tutorial”, spark_file=”spark.py”, additional_python_files=[“additional.py”])
Options that can be passed to the bqsparkstoredprocedure method are: - name: name of resource - dataset_id: dataset id where the routine will be created - func (optional): function/method to be executed - runtime_version (optional): runtime version of the spark code - container_image (optional): container image to use - spark_file (optional): file from local path with the spark code - additional_python_files (optional): List of files from local path with additional code (Ex: libraries) - additional_files (optional): List of files from local path with additional files (Ex: csvs) - properties (optional): Dictionary with additional properties. Supported properties