.. _infrastructure:

================
Infrastructure
================

You can  provision infrastructure within your Goblet code.

.. _redis:

Redis
^^^^^

.. code:: python

    app = Goblet()
    app.redis("redis-example")

When deploying a backend, the environment variables will automatically be updated to include the `REDIS_INSTANCE_NAME`, `REDIS_HOST` and `REDIS_PORT`. 
To further configure your Redis Instance within Goblet, specify the **`redis`** key in your `config.json`. 
You can reference `Redis Instance Resource <https://cloud.google.com/memorystore/docs/redis/reference/rest/v1/projects.locations.instances#Instance>`_ for more information on available fields.

VPC Connector
^^^^^^^^^^^^^
.. code:: python

    app = Goblet()
    app.vpcconnector("vpcconnector")

When deploying a backend, the vpc access configuration will be updated to include the specified vpc connector.
To further configure your VPC Connector within Goblet, specify the **`vpcconnector`** key in your `config.json`. 
You can reference `Connector Resource <https://cloud.google.com/vpc/docs/reference/vpcaccess/rest/v1/projects.locations.connectors#Connector>`_  for more information on available fields.

.. note::
    * In order to ensure proper configuration of the VPC Connector, the `ipCidrRange` key is required to be set within `vpcconnector` of your `config.json`.

.. _apigateway:

Api Gateway
^^^^^^^^^^^^^

.. code:: python

    app = Goblet(function_name="openapi-existing")
    
    filename = "openapi_spec.yml"
    app.apigateway("openapi-existing", "BACKEND_URL", filename=filename)

.. code:: python 

    app = Goblet(function_name="openapi-existing-dict")

    openapi_dict = {
            "swagger": "2.0",
            "info": {
                "title": "media-serving-service",
                "description": "Goblet Autogenerated Spec",
                "version": "1.0.0",
            },
            "schemes": ["https"],
            "produces": ["application/json"],
            "paths": {
                "/": {
                    "get": {
                        "operationId": "get_main",
                        "responses": {"200": {"description": "A successful response"}},
                    }
                }
            },
            "definitions": {},
        }
    app.apigateway("openapi-existing-from-dict", "BACKEND_URL", openapi_dict=openapi_dict)

You can deploy an Api Gateway and all related resources (Api, Api Config, Gateway) with an existing openapi spec using the apigateway decorator. The decorator can take in a filename or the 
spec as a dictionary. If you dont need to deploy any other Goblet resources you can deploy just the Api Gateway using 
`goblet deploy -p PROJECT --skip-backend --skip-resources`

By default there is a timeout on Api Gateway of 15 seconds. This can be overriden by setting `"api_gateway": {"deadline": 45}` in `config.json`. 

.. note::
    
    * API Gateway only supports swagger 2.0. 

CloudTask Queue
^^^^^^^^^^^^^^^
.. code:: python

    app = Goblet()
    config = { ... }
    client = app.cloudtaskqueue("queue", config=config)

To further configure your CloudTaskQueue within Goblet, provide the config parameter base on the documentation. `CloudTask Queue Resource <https://cloud.google.com/tasks/docs/reference/rest/v2/projects.locations.queues#Queue>`_.
The configuration can be provided inline when declaring the queue, or in your config.json under the **cloudtaskqueue** key.

.. code::

    {
        "cloudtaskqueue": {
            "queue": { ... }
        }
    }

PubSub Topics
^^^^^^^^^^^^^

.. code:: python

    app = Goblet()
    config = { ... }
    app.pubsub_topic("topic", config=config)

To further configure your PubSub topic within Goblet, provide the config parameter base on the documentation. `Topic Resource <https://cloud.google.com/pubsub/docs/reference/rest/v1/projects.topic>`_.

BigQuery Spark Stored Procedures
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^


To deploy BigQuery stored procedures using Spark follow the example below. 
BigQuery stored procedures documentation can be found `here <https://cloud.google.com/bigquery/docs/spark-procedures>`_.

Using a function from the same python file:
.. code:: python

    import logging
    from goblet import Goblet, goblet_entrypoint
    import pyspark.sql.functions as F
    from pyspark.sql import SparkSession

    app = Goblet(function_name="create-bq-spark-stored-procedure")

    app.log.setLevel(logging.DEBUG)  # configure goblet logger level
    goblet_entrypoint(app)

    # Create a bq spark stored procedure with the spark code and additional python files
    def spark_handler():

        spark = SparkSession.builder.appName("spark-bigquery-demo").getOrCreate()

        # Load data from BigQuery.
        texts = spark.read.format("bigquery") \
        .option("table", "tutorial.poc") \
        .load()
        texts.createOrReplaceTempView("words")

        # Perform word count.
        text_count = texts.select("id", "text", F.length("text").alias("sum_text_count"))
        text_count.show()
        text_count.printSchema()

        # Saving the data to BigQuery
        text_count.write.mode("append").format("bigquery") \
        .option("writeMethod", "direct") \
        .save("tutorial.wordcount_output")

    app.bqsparkstoredprocedure(
        name="count_words_procedure_external",
        dataset_id="tutorial",
        func=spark_handler)

Using a function from a different python file and loading additional python files:
`spark.py`:
.. code:: python

    def spark_handler():
        from pyspark.sql import SparkSession
        import pyspark.sql.functions as F
        spark = SparkSession.builder.appName("spark-bigquery-demo").getOrCreate()

        # Load data from BigQuery.
        texts = spark.read.format("bigquery") \
        .option("table", "tutorial.poc") \
        .load()
        texts.createOrReplaceTempView("words")

        # Perform word count.
        text_count = texts.select("id", "text", F.length("text").alias("sum_text_count"))
        text_count.show()
        text_count.printSchema()

        # Saving the data to BigQuery
        text_count.write.mode("append").format("bigquery") \
        .option("writeMethod", "direct") \
        .save("tutorial.wordcount_output")

    if __name__ == "__main__":
        spark_handler()

`additional.py`:
.. code:: python

    import logging
    logging.basicConfig(level=logging.INFO)
    logger = logging.getLogger(__name__)

    def additional_func():
        logger.info("additional_func")

`main.py`:
.. code:: python

    import logging
    from goblet import Goblet, goblet_entrypoint

    app = Goblet(function_name="create-bq-spark-stored-procedure")

    app.log.setLevel(logging.DEBUG)  # configure goblet logger level
    goblet_entrypoint(app)

    # Create a bq spark stored procedure with the spark code and additional python files
    app.bqsparkstoredprocedure(name="count_words_procedure_external", dataset_id="tutorial", spark_file="spark.py", additional_python_files=["additional.py"])


Options that can be passed to the `bqsparkstoredprocedure` method are:
- name: name of resource
- dataset_id: dataset id where the routine will be created
- func (optional): function/method to be executed
- runtime_version (optional): runtime version of the spark code
- container_image (optional): container image to use
- spark_file (optional): file from local path with the spark code
- additional_python_files (optional): List of files from local path with additional code (Ex: libraries)
- additional_files (optional): List of files from local path with additional files (Ex: csvs)
- properties (optional): Dictionary with additional properties. `Supported properties <https://spark.apache.org/docs/latest/configuration.html#spark-properties>`_