{ "cells": [ { "cell_type": "markdown", "id": "7f1f5a4a-ae34-4a27-9efa-399edc0e384a", "metadata": { "tags": [] }, "source": [ "## Benchmark: ClickHouse Vs. InfluxDB Vs. Postgresql Vs. Parquet \n", "\n", "-----\n", "\n", "#### How to use:\n", "* Rename the file \"properties-model.ini\" to \"properties.ini\"\n", "* Fill with your own credentials\n", "----\n", "\n", "The proposal of this work is to compare the speed in read/writing a midle level of data ( a dataset with 9 columns and 50.000 lines) to four diferent databases:\n", "* ClickHouse\n", "* InfluxDB\n", "* Postgresql\n", "* Parquet (in a S3 Minio Storage)
\n", "ToDo:
\n", "* DuckDB with Polars\n", "* MongoDB\n", "* Kdb+\n", "\n", " \n", "Deve-se relevar:\n", "é uma \"cold-storage\" ou \"frezze-storage\"?
\n", "influxdb: alta leitura e possui a vantagem da indexaçõa para vizualização de dados em gráficos.\n", "\n", "notas: \n", "* comparar tamanho do csv com parquet" ] }, { "cell_type": "markdown", "id": "6bb26ce7-1e84-4665-accd-916bb977f95d", "metadata": { "tags": [] }, "source": [ "### Imports " ] }, { "cell_type": "code", "execution_count": 11, "id": "ab6c6c81-6ac1-4668-a79b-a9a0341fb35a", "metadata": { "tags": [] }, "outputs": [ { "data": { "text/plain": [ "False" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import configparser\n", "import io\n", "import timeit\n", "from datetime import datetime\n", "\n", "import duckdb\n", "\n", "# from influxdb_client import InfluxDBClient\n", "import influxdb_client\n", "import matplotlib.pyplot as plt\n", "import numpy as np\n", "import pandas as pd\n", "import pdmongo as pdm\n", "from clickhouse_driver import Client\n", "from dotenv import load_dotenv\n", "\n", "# from influxdb_client.client.write_api import SYNCHRONOUS\n", "from minio import Minio\n", "from pymongo import MongoClient\n", "from pytz import timezone\n", "from qpython import qconnection\n", "from sqlalchemy import create_engine\n", "\n", "load_dotenv()" ] }, { "cell_type": "code", "execution_count": 2, "id": "55c3cd57-0996-4723-beb5-8f3196c96009", "metadata": { "tags": [] }, "outputs": [], "source": [ "# Variables\n", "dbname = \"EURUSDtest\"\n", "collection = \"finance\"" ] }, { "cell_type": "code", "execution_count": 3, "id": "968403e3-2e5e-4834-b969-be4600e2963a", "metadata": { "tags": [] }, "outputs": [], "source": [ "arq = configparser.RawConfigParser()\n", "arq.read(\"properties.ini\")\n", "ClickHouseUser = arq.get(\"CLICKHOUSE\", \"user\")\n", "ClickHouseKey = arq.get(\"CLICKHOUSE\", \"key\")\n", "ClickHouseUrl = arq.get(\"CLICKHOUSE\", \"url\")\n", "\n", "InfluxDBUser = arq.get(\"INFLUXDB\", \"user\")\n", "InfluxDBKey = arq.get(\"INFLUXDB\", \"key\")\n", "InfluxDBUrl = arq.get(\"INFLUXDB\", \"url\")\n", "InfluxDBBucket = arq.get(\"INFLUXDB\", \"bucket\")\n", "\n", "PostgresqlUser = arq.get(\"POSTGRESQL\", \"user\")\n", "PostgresqlKey = arq.get(\"POSTGRESQL\", \"key\")\n", "PostgresqlUrl = arq.get(\"POSTGRESQL\", \"url\")\n", "PostgresqlDB = arq.get(\"POSTGRESQL\", \"database\")\n", "\n", "S3MinioUser = arq.get(\"S3MINIO\", \"user\")\n", "S3MinioKey = arq.get(\"S3MINIO\", \"key\")\n", "S3MinioUrl = arq.get(\"S3MINIO\", \"url\")\n", "S3MinioRegion = arq.get(\"S3MINIO\", \"region\")\n", "\n", "MongoUser = arq.get(\"MONGODB\", \"user\")\n", "MongoKey = arq.get(\"MONGODB\", \"key\")\n", "MongoUrl = arq.get(\"MONGODB\", \"url\")" ] }, { "cell_type": "code", "execution_count": 4, "id": "3634a4ec-04c2-4f1e-8659-5d22eb17a254", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
idfromattoopencloseminmaxvolume
99999579847482023-03-03 18:13:3016778672250000000002023-03-03 18:13:451.0626951.0626351.0626301.06270064
99999679847492023-03-03 18:13:4516778672400000000002023-03-03 18:14:001.0626451.0626501.0626251.06265043
99999779847502023-03-03 18:14:0016778672550000000002023-03-03 18:14:151.0626401.0626251.0626201.06266547
99999879847512023-03-03 18:14:1516778672700000000002023-03-03 18:14:301.0626251.0625351.0625351.06264543
99999979847522023-03-03 18:14:3016778672850000000002023-03-03 18:14:451.0625351.0625201.0625201.06258059
\n", "
" ], "text/plain": [ " id from at \\\n", "999995 7984748 2023-03-03 18:13:30 1677867225000000000 \n", "999996 7984749 2023-03-03 18:13:45 1677867240000000000 \n", "999997 7984750 2023-03-03 18:14:00 1677867255000000000 \n", "999998 7984751 2023-03-03 18:14:15 1677867270000000000 \n", "999999 7984752 2023-03-03 18:14:30 1677867285000000000 \n", "\n", " to open close min max volume \n", "999995 2023-03-03 18:13:45 1.062695 1.062635 1.062630 1.062700 64 \n", "999996 2023-03-03 18:14:00 1.062645 1.062650 1.062625 1.062650 43 \n", "999997 2023-03-03 18:14:15 1.062640 1.062625 1.062620 1.062665 47 \n", "999998 2023-03-03 18:14:30 1.062625 1.062535 1.062535 1.062645 43 \n", "999999 2023-03-03 18:14:45 1.062535 1.062520 1.062520 1.062580 59 " ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# %%time\n", "# Load Dataset\n", "df = pd.read_csv(\"out.csv\", index_col=0)\n", "df.tail()" ] }, { "cell_type": "code", "execution_count": null, "id": "76199f91-31d6-416b-9f15-5d435b3792c9", "metadata": {}, "outputs": [], "source": [ "df[\"from\"] = pd.to_datetime(df[\"from\"], unit=\"s\")\n", "df[\"to\"] = pd.to_datetime(df[\"to\"], unit=\"s\")\n", "# Optional use when not transoformed yet\n", "# Transform Datetime" ] }, { "cell_type": "markdown", "id": "274cc026-2f48-4e38-b80f-b1a9ff982060", "metadata": { "tags": [] }, "source": [ "#### Funçoes\n", "\n", "-> Class" ] }, { "cell_type": "code", "execution_count": 5, "id": "27de1ec8-4de1-440a-b555-b4a46c5ef7ce", "metadata": {}, "outputs": [], "source": [ "def timestamp2dataHora(x, timezone_=\"America/Sao_Paulo\"):\n", " d = datetime.fromtimestamp(x, tz=timezone(timezone_))\n", " return d" ] }, { "cell_type": "markdown", "id": "4a8d5703-9bc9-4d38-83ff-457159304d58", "metadata": { "jp-MarkdownHeadingCollapsed": true, "tags": [] }, "source": [ "### ClickHouse" ] }, { "cell_type": "code", "execution_count": 8, "id": "c3202bbb-2655-45b2-b166-9f45a3ef854c", "metadata": { "tags": [] }, "outputs": [ { "data": { "text/plain": [ "'Database created'" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# !! driver tcp.\n", "def cHouseConnect():\n", " client = Client(\n", " host=ClickHouseUrl,\n", " user=ClickHouseUser,\n", " password=ClickHouseKey,\n", " settings={\"use_numpy\": True},\n", " )\n", " return client\n", "\n", "\n", "# Create Tables in ClickHouse\n", "# !! ALTERAR TIPOS !!\n", "# ENGINE: 'Memory' desaparece quando server é reiniciado\n", "def cHouseCreateDb(databasename):\n", " client = cHouseConnect()\n", " client.execute(\n", " \"CREATE TABLE IF NOT EXISTS {} (id UInt32,\"\n", " \"from DateTime, at UInt64, to DateTime, open Float64,\"\n", " \"close Float64, min Float64, max Float64, volume UInt32)\"\n", " \"ENGINE MergeTree ORDER BY to\".format(databasename)\n", " )\n", " client.disconnect()\n", " return \"Database created\"\n", "\n", "\n", "# Write dataframe to db\n", "def cHouseInsertDf(dbName, dataframe):\n", " client = cHouseConnect()\n", " client.insert_dataframe(\"INSERT INTO {} VALUES\".format(dbName), dataframe)\n", " client.disconnect()\n", " return \" dataframe {} inserted in clickhouse database\".format(dataframe)\n", "\n", "\n", "def cHouseQueryDf(databaseName):\n", " client = cHouseConnect()\n", " dfQuery = client.query_dataframe(\"SELECT * FROM default.{}\".format(databaseName))\n", " client.disconnect()\n", " return dfQuery\n", "\n", "\n", "cHouseCreateDb(dbname)" ] }, { "cell_type": "code", "execution_count": 9, "id": "cc4865b3-a1bc-4a35-9624-15334754b3a1", "metadata": {}, "outputs": [], "source": [ "# Insert to db and benchmark time\n", "start = timeit.default_timer()\n", "cHouseInsertDf(dbname, df)\n", "stop = timeit.default_timer()\n", "cHouse_write_execution_time = stop - start" ] }, { "cell_type": "code", "execution_count": 10, "id": "1fac82c1-2d04-44ef-893a-dc13b755e6d8", "metadata": {}, "outputs": [], "source": [ "# read from db and benchmark time\n", "start = timeit.default_timer()\n", "dfCh = cHouseQueryDf(dbname)\n", "stop = timeit.default_timer()\n", "cHouse_read_execution_time = stop - start" ] }, { "cell_type": "code", "execution_count": 11, "id": "597ae7bd-2eea-44d7-b379-f0eb7e745c15", "metadata": { "tags": [] }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
idfromattoopencloseminmaxvolume
99999578577742023-02-01 17:06:0016752711750000000002023-02-01 17:06:151.0917251.0916701.091661.0917684
99999678577752023-02-01 17:06:1516752711900000000002023-02-01 17:06:301.0916801.0916601.091651.0916851
99999778577752023-02-01 17:06:1516752711900000000002023-02-01 17:06:301.0916801.0916601.091651.0916851
99999878577762023-02-01 17:06:3016752712050000000002023-02-01 17:06:451.0916601.0916551.091641.0916863
99999978577762023-02-01 17:06:3016752712050000000002023-02-01 17:06:451.0916601.0916551.091641.0916863
\n", "
" ], "text/plain": [ " id from at to \\\n", "999995 7857774 2023-02-01 17:06:00 1675271175000000000 2023-02-01 17:06:15 \n", "999996 7857775 2023-02-01 17:06:15 1675271190000000000 2023-02-01 17:06:30 \n", "999997 7857775 2023-02-01 17:06:15 1675271190000000000 2023-02-01 17:06:30 \n", "999998 7857776 2023-02-01 17:06:30 1675271205000000000 2023-02-01 17:06:45 \n", "999999 7857776 2023-02-01 17:06:30 1675271205000000000 2023-02-01 17:06:45 \n", "\n", " open close min max volume \n", "999995 1.091725 1.091670 1.09166 1.09176 84 \n", "999996 1.091680 1.091660 1.09165 1.09168 51 \n", "999997 1.091680 1.091660 1.09165 1.09168 51 \n", "999998 1.091660 1.091655 1.09164 1.09168 63 \n", "999999 1.091660 1.091655 1.09164 1.09168 63 " ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dfCh.tail()" ] }, { "cell_type": "code", "execution_count": 12, "id": "86794e47-611f-4ca8-a7e8-07e71afafe67", "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "5.7032926019999195\n" ] } ], "source": [ "print(cHouse_read_execution_time)" ] }, { "cell_type": "code", "execution_count": 13, "id": "e7926062-8e84-4d3f-90a9-32807ce4f3d4", "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "6.242225094000105\n" ] } ], "source": [ "print(cHouse_write_execution_time)" ] }, { "cell_type": "code", "execution_count": 28, "id": "8faa5683-a204-461d-80c3-67644aa714ce", "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "CPU times: user 2.21 s, sys: 383 ms, total: 2.6 s\n", "Wall time: 10.7 s\n" ] } ], "source": [ "# %%time\n", "# dfCh = cHouseQueryDf(dbname)" ] }, { "cell_type": "markdown", "id": "1d389546-911f-43f7-aad1-49f7bcc83503", "metadata": { "tags": [] }, "source": [ "### InfluxDB\n" ] }, { "cell_type": "code", "execution_count": 33, "id": "ecd217ab-0e16-40a6-9b92-9212b9bb20e9", "metadata": { "tags": [] }, "outputs": [], "source": [ "query = \"\"\"\n", "from(bucket: \"EURUSDtest\")\n", "|> range(start:2023-03-03T18:14:30Z, stop: now())\n", "|> filter(fn: (r) => r._measurement == \"id\")\n", "|> pivot(rowKey:[\"_time\"], columnKey: [\"_field\"], valueColumn: \"_value\")\"\"\"" ] }, { "cell_type": "code", "execution_count": 34, "id": "c3e7ebfd-76f1-4ac4-9833-312eb1a531af", "metadata": {}, "outputs": [], "source": [ "def influxdbConnect():\n", " client = influxdb_client.InfluxDBClient(\n", " url=InfluxDBUrl, token=InfluxDBKey, org=InfluxDBUser\n", " )\n", " return client\n", "\n", "\n", "def influxdbLoadCsv(csv=\"out.csv\", dictDates=[\"from\", \"to\"], index=\"from\"):\n", " # Read data from CSV without index and parse 'TimeStamp' as date.\n", " df = pd.read_csv(csv, sep=\",\", index_col=False, parse_dates=dictDates)\n", " # Set 'TimeStamp' field as index of dataframe # test another indexs\n", " df.set_index(index, inplace=True)\n", " return df\n", "\n", "\n", "def influxdbWriteCsv(dataFrame, bucket, measurement=\"id\", tag=\"volume\"):\n", " client = influxdbConnect()\n", " # write_options=SYNCHRONOUS\n", " with client.write_api() as writer:\n", " writer.write(\n", " bucket=bucket,\n", " record=dataFrame,\n", " data_frame_measurement_name=measurement,\n", " data_frame_tag_columns=[tag],\n", " )\n", " writer.__del__()\n", " client.__del__()\n", " return 0\n", "\n", "\n", "def influxdRead(org=InfluxDBUrl, query=query):\n", " client = influxdbConnect()\n", " InfluxDf = client.query_api().query_data_frame(org=\"librography\", query=query)\n", " # display(InfluxDf.head())\n", " return InfluxDf" ] }, { "cell_type": "code", "execution_count": 96, "id": "e05266b8-ff32-462c-b059-325a40a53d25", "metadata": { "tags": [] }, "outputs": [], "source": [ "dafr = influxdbLoadCsv()\n", "# dafr.head()" ] }, { "cell_type": "code", "execution_count": null, "id": "95191283-f11e-456f-8395-36981ab1ac51", "metadata": {}, "outputs": [], "source": [ "start = timeit.default_timer()\n", "influxdbWriteCsv(dafr, InfluxDBBucket)\n", "# dfIdw = cHouseQueryDf(dbname)\n", "stop = timeit.default_timer()\n", "influxdb_write_execution_time = stop - start" ] }, { "cell_type": "code", "execution_count": 119, "id": "3e5bb029-4988-4692-bc5f-9a6f7fcc2159", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "161.51344409900048\n" ] } ], "source": [ "print(influxdb_write_execution_time)" ] }, { "cell_type": "code", "execution_count": 120, "id": "850c6921-5e1c-417a-bea6-ea18be642008", "metadata": { "tags": [] }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
resulttable_start_stop_time_measurementvolumeUnnamed: 0atcloseidmaxminopento
0_result02023-03-03 18:14:30+00:002023-06-17 02:47:50.721233+00:002023-03-05 22:01:00+00:00id011558916780536750000000001.06342579856541.0634251.0634251.0634252023-03-05 22:01:15
1_result02023-03-03 18:14:30+00:002023-06-17 02:47:50.721233+00:002023-03-05 23:58:30+00:00id011605916780607250000000001.06259579861241.0625951.0625951.0625952023-03-05 23:58:45
2_result02023-03-03 18:14:30+00:002023-06-17 02:47:50.721233+00:002023-03-06 23:58:30+00:00id012181916781471250000000001.06861579918841.0686151.0686151.0686152023-03-06 23:58:45
3_result02023-03-03 18:14:30+00:002023-06-17 02:47:50.721233+00:002023-03-06 23:59:30+00:00id012182316781471850000000001.06860579918881.0686051.0686051.0686052023-03-06 23:59:45
4_result02023-03-03 18:14:30+00:002023-06-17 02:47:50.721233+00:002023-03-08 23:59:00+00:00id018249316783199550000000001.05489580034061.0548951.0548951.0548952023-03-08 23:59:15
\n", "
" ], "text/plain": [ " result table _start _stop \\\n", "0 _result 0 2023-03-03 18:14:30+00:00 2023-06-17 02:47:50.721233+00:00 \n", "1 _result 0 2023-03-03 18:14:30+00:00 2023-06-17 02:47:50.721233+00:00 \n", "2 _result 0 2023-03-03 18:14:30+00:00 2023-06-17 02:47:50.721233+00:00 \n", "3 _result 0 2023-03-03 18:14:30+00:00 2023-06-17 02:47:50.721233+00:00 \n", "4 _result 0 2023-03-03 18:14:30+00:00 2023-06-17 02:47:50.721233+00:00 \n", "\n", " _time _measurement volume Unnamed: 0 \\\n", "0 2023-03-05 22:01:00+00:00 id 0 115589 \n", "1 2023-03-05 23:58:30+00:00 id 0 116059 \n", "2 2023-03-06 23:58:30+00:00 id 0 121819 \n", "3 2023-03-06 23:59:30+00:00 id 0 121823 \n", "4 2023-03-08 23:59:00+00:00 id 0 182493 \n", "\n", " at close id max min open \\\n", "0 1678053675000000000 1.063425 7985654 1.063425 1.063425 1.063425 \n", "1 1678060725000000000 1.062595 7986124 1.062595 1.062595 1.062595 \n", "2 1678147125000000000 1.068615 7991884 1.068615 1.068615 1.068615 \n", "3 1678147185000000000 1.068605 7991888 1.068605 1.068605 1.068605 \n", "4 1678319955000000000 1.054895 8003406 1.054895 1.054895 1.054895 \n", "\n", " to \n", "0 2023-03-05 22:01:15 \n", "1 2023-03-05 23:58:45 \n", "2 2023-03-06 23:58:45 \n", "3 2023-03-06 23:59:45 \n", "4 2023-03-08 23:59:15 " ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# read from db and benchmark time\n", "start = timeit.default_timer()\n", "influxdRead()\n", "stop = timeit.default_timer()\n", "influxdb_read_execution_time = stop - start" ] }, { "cell_type": "code", "execution_count": 121, "id": "3ee3c0dd-cb70-4124-a0fb-db8dd2c134c0", "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "181.9998163249984\n" ] } ], "source": [ "print(influxdb_read_execution_time)" ] }, { "cell_type": "code", "execution_count": 19, "id": "54342a28-ba2b-4ade-a692-00566b53a639", "metadata": { "tags": [] }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Unnamed: 0idattoopencloseminmaxvolume
from
2023-03-03 18:13:30999995798474816778672250000000002023-03-03 18:13:451.0626951.0626351.0626301.06270064
2023-03-03 18:13:45999996798474916778672400000000002023-03-03 18:14:001.0626451.0626501.0626251.06265043
2023-03-03 18:14:00999997798475016778672550000000002023-03-03 18:14:151.0626401.0626251.0626201.06266547
2023-03-03 18:14:15999998798475116778672700000000002023-03-03 18:14:301.0626251.0625351.0625351.06264543
2023-03-03 18:14:30999999798475216778672850000000002023-03-03 18:14:451.0625351.0625201.0625201.06258059
\n", "
" ], "text/plain": [ " Unnamed: 0 id at \\\n", "from \n", "2023-03-03 18:13:30 999995 7984748 1677867225000000000 \n", "2023-03-03 18:13:45 999996 7984749 1677867240000000000 \n", "2023-03-03 18:14:00 999997 7984750 1677867255000000000 \n", "2023-03-03 18:14:15 999998 7984751 1677867270000000000 \n", "2023-03-03 18:14:30 999999 7984752 1677867285000000000 \n", "\n", " to open close min \\\n", "from \n", "2023-03-03 18:13:30 2023-03-03 18:13:45 1.062695 1.062635 1.062630 \n", "2023-03-03 18:13:45 2023-03-03 18:14:00 1.062645 1.062650 1.062625 \n", "2023-03-03 18:14:00 2023-03-03 18:14:15 1.062640 1.062625 1.062620 \n", "2023-03-03 18:14:15 2023-03-03 18:14:30 1.062625 1.062535 1.062535 \n", "2023-03-03 18:14:30 2023-03-03 18:14:45 1.062535 1.062520 1.062520 \n", "\n", " max volume \n", "from \n", "2023-03-03 18:13:30 1.062700 64 \n", "2023-03-03 18:13:45 1.062650 43 \n", "2023-03-03 18:14:00 1.062665 47 \n", "2023-03-03 18:14:15 1.062645 43 \n", "2023-03-03 18:14:30 1.062580 59 " ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.tail()" ] }, { "cell_type": "markdown", "id": "b9ddfdc6-c899-4f6c-9b4e-8ec6ab6d7e05", "metadata": { "jp-MarkdownHeadingCollapsed": true, "tags": [] }, "source": [ "### Postgresql" ] }, { "cell_type": "code", "execution_count": 24, "id": "16cd8eb7-333d-43fd-88e0-ee983645d3fd", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Engine(postgresql+psycopg2://postgres:***@192.168.1.133:5432/postgres)" ] }, "execution_count": 24, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Connect / Create Tables\n", "def psqlConnect():\n", " engine = create_engine(\n", " \"postgresql+psycopg2://{}:{}@{}:5432/{}\".format(\n", " PostgresqlUser, PostgresqlKey, PostgresqlUrl, PostgresqlDB\n", " )\n", " )\n", " return engine\n", "\n", "\n", "psqlConnect()\n", "# testar função" ] }, { "cell_type": "code", "execution_count": 25, "id": "be31f3a0-b7ed-48e6-9b65-dc16319fb8d1", "metadata": {}, "outputs": [], "source": [ "# Drop old table and create new empty table\n", "def psqlCreateTables(databaseName):\n", " engine = psqlConnect()\n", " df.head(0).to_sql(databaseName, engine, if_exists=\"replace\", index=False)\n", " # Write\n", " conn = engine.raw_connection()\n", " cur = conn.cursor()\n", " output = io.StringIO()\n", " df.to_csv(output, sep=\"\\t\", header=False, index=False)\n", " output.seek(0)\n", " contents = output.getvalue()\n", "\n", " cur.copy_from(output, \"comparedbs\") # , null=\"\") # null values become ''\n", " conn.commit()\n", " cur.close()\n", " conn.close()\n", " # disconnect()\n", " return 0\n", "\n", "\n", "# funcao read sql\n", "def psqlReadTables():\n", " engine = psqlConnect()\n", " df = pd.read_sql_query('select * from \"comparedbs\"', con=engine)\n", " return df\n", "\n", "\n", "# testar função" ] }, { "cell_type": "code", "execution_count": 26, "id": "98cc9360-4b84-43e4-b23b-b32d0c50c3b9", "metadata": { "tags": [] }, "outputs": [], "source": [ "# Insert to db and benchmark time\n", "start = timeit.default_timer()\n", "psqlCreateTables(dbname)\n", "stop = timeit.default_timer()\n", "psql_write_execution_time = stop - start" ] }, { "cell_type": "code", "execution_count": 27, "id": "82e1b44e-35af-403f-9936-5e1561fd5abf", "metadata": { "tags": [] }, "outputs": [], "source": [ "start = timeit.default_timer()\n", "psqlReadTables()\n", "stop = timeit.default_timer()\n", "psql_read_execution_time = stop - start" ] }, { "cell_type": "code", "execution_count": 29, "id": "6d1b7480-5bc7-4f08-8cf3-b9590802d8f7", "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "126.40752380799677\n" ] } ], "source": [ "print(psql_read_execution_time)" ] }, { "cell_type": "code", "execution_count": null, "id": "6acb2959-3255-43bd-aea5-9ef70acc8902", "metadata": { "tags": [] }, "outputs": [], "source": [ "# df.head()" ] }, { "cell_type": "markdown", "id": "f9e0393d-7d1d-406a-a068-9dbf4968e977", "metadata": { "jp-MarkdownHeadingCollapsed": true, "tags": [] }, "source": [ "### S3 Parquet" ] }, { "cell_type": "code", "execution_count": 15, "id": "7c7022bf-9c3b-400a-9045-b089483f05ad", "metadata": { "tags": [] }, "outputs": [], "source": [ "# fazer sem funçao para ver se melhora\n", "# verifique se esta no ssd os arquivos da pasta git\n", "def s3Connect():\n", " client = Minio(\n", " S3MinioUrl,\n", " secure=False,\n", " region=S3MinioRegion,\n", " access_key=S3MinioUser,\n", " secret_key=S3MinioKey,\n", " )\n", " return client\n", "\n", "\n", "def s3CreateBucket(bucketName=\"data\"):\n", " client = s3Connect()\n", " found = client.bucket_exists(bucketName)\n", " if not found:\n", " return client.make_bucket(bucketName)\n", " else:\n", " return \"Bucket '{}' already exists\".format(bucketName)\n", "\n", "\n", "def s3uploadCsv():\n", " client = s3Connect()\n", " client.fput_object(\n", " \"data\",\n", " \"data.parquet\",\n", " \"data/data.parquet\",\n", " )\n", " return (\n", " \"'data/data.parquet' is successfully uploaded as \"\n", " \"object 'data.parquet' to bucket 'data'.\"\n", " )" ] }, { "cell_type": "code", "execution_count": 13, "id": "cd7fe012-9eee-4f91-8c07-8e0148633766", "metadata": { "tags": [] }, "outputs": [], "source": [ "# Insert to db and benchmark time\n", "df.to_parquet(\"data/data.parquet\")\n", "s3CreateBucket()\n", "start = timeit.default_timer()\n", "s3uploadCsv()\n", "stop = timeit.default_timer()\n", "s3_write_execution_time = stop - start" ] }, { "cell_type": "code", "execution_count": null, "id": "a803e870-1970-496f-b678-a7f8866870cf", "metadata": {}, "outputs": [], "source": [ "# falta read (parquet to df)" ] }, { "cell_type": "code", "execution_count": 14, "id": "390918c8-c88f-404a-96c4-685d578fdad0", "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "3.8389689489995362\n" ] } ], "source": [ "print(s3_write_execution_time)" ] }, { "cell_type": "code", "execution_count": 16, "id": "666d4bc7-933d-4e6e-87c1-72fd37e61e0d", "metadata": { "tags": [] }, "outputs": [], "source": [ "start = timeit.default_timer()\n", "pq = pd.read_parquet(\"data/data.parquet\", engine=\"pyarrow\")\n", "stop = timeit.default_timer()\n", "s3_read_execution_time = stop - start" ] }, { "cell_type": "code", "execution_count": 17, "id": "573b10c3-5dfe-420a-a988-688d4ebcde24", "metadata": { "tags": [] }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
idfromattoopencloseminmaxvolume
077308012023-01-02 15:58:4516726751400000000002023-01-02 15:59:001.0659951.0660351.0659301.06607057
177308022023-01-02 15:59:0016726751550000000002023-01-02 15:59:151.0660551.0660851.0660051.06611552
277308032023-01-02 15:59:1516726751700000000002023-01-02 15:59:301.0660801.0660251.0660251.06611057
377308042023-01-02 15:59:3016726751850000000002023-01-02 15:59:451.0659801.0659851.0658851.06604564
477308052023-01-02 15:59:4516726752000000000002023-01-02 16:00:001.0659751.0660551.0658301.06605550
\n", "
" ], "text/plain": [ " id from at to \\\n", "0 7730801 2023-01-02 15:58:45 1672675140000000000 2023-01-02 15:59:00 \n", "1 7730802 2023-01-02 15:59:00 1672675155000000000 2023-01-02 15:59:15 \n", "2 7730803 2023-01-02 15:59:15 1672675170000000000 2023-01-02 15:59:30 \n", "3 7730804 2023-01-02 15:59:30 1672675185000000000 2023-01-02 15:59:45 \n", "4 7730805 2023-01-02 15:59:45 1672675200000000000 2023-01-02 16:00:00 \n", "\n", " open close min max volume \n", "0 1.065995 1.066035 1.065930 1.066070 57 \n", "1 1.066055 1.066085 1.066005 1.066115 52 \n", "2 1.066080 1.066025 1.066025 1.066110 57 \n", "3 1.065980 1.065985 1.065885 1.066045 64 \n", "4 1.065975 1.066055 1.065830 1.066055 50 " ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "pq.head()" ] }, { "cell_type": "code", "execution_count": 18, "id": "fefb54f6-9040-4fd7-9861-20322c202d92", "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0.5066086639999412\n" ] } ], "source": [ "print(s3_read_execution_time)" ] }, { "cell_type": "markdown", "id": "50d1fc58-89a7-4507-aff0-6e943656cfe0", "metadata": { "jp-MarkdownHeadingCollapsed": true, "tags": [] }, "source": [ "### MongoDB" ] }, { "cell_type": "code", "execution_count": 71, "id": "d104d9af-fa34-4261-8478-329a28ee4f2e", "metadata": { "tags": [] }, "outputs": [], "source": [ "port = \"27017\"\n", "\n", "\n", "def mongoLoadCsv(csvfile):\n", " data = pd.read_csv(\"out.csv\")\n", " return data\n", "\n", "\n", "def mongoConnect():\n", " client = MongoClient(\n", " \"mongodb://{}:{}@{}/EURUSDtest?retryWrites=true&w=majority\".format(\n", " MongoUser, MongoKey, MongoUrl\n", " ),\n", " authSource=\"admin\",\n", " )\n", " return client\n", "\n", "\n", "def mongoWriteDict(dados, dbs, collection):\n", " client = mongoConnect()\n", " db = client[dbs]\n", " collection = db[collection]\n", " # data.reset_index(inplace=True)\n", " data_dict = dados.to_dict(\"records\")\n", " # return data_dict\n", " collection.insert_many(data_dict)\n", " return 0\n", "\n", "\n", "def mongoRead():\n", " df = pdm.read_mongo(\n", " collection, [], \"mongodb://192.168.1.162:27017/{}\".format(dbname)\n", " )\n", " return df" ] }, { "cell_type": "code", "execution_count": null, "id": "739de6aa-313f-4ccd-96c8-fa22d0cc687e", "metadata": { "tags": [] }, "outputs": [], "source": [ "data = mongoLoadCsv(\"out.csv\")\n", "data.head()" ] }, { "cell_type": "code", "execution_count": 51, "id": "0af8f72c-5b58-4dfc-af36-c5b4bc79f127", "metadata": { "tags": [] }, "outputs": [], "source": [ "start = timeit.default_timer()\n", "dfCh = mongoWriteDict(data, dbname, \"finance\")\n", "stop = timeit.default_timer()\n", "mongo_write_execution_time = stop - start" ] }, { "cell_type": "code", "execution_count": 52, "id": "0757f14c-4677-41d3-90d8-63b884e24e7e", "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "46.76343438199547\n" ] } ], "source": [ "print(mongo_write_execution_time)" ] }, { "cell_type": "code", "execution_count": 72, "id": "e7922312-74cb-4df3-8dea-e5ee0d99fab7", "metadata": { "tags": [] }, "outputs": [], "source": [ "start = timeit.default_timer()\n", "dfMongo = mongoRead()\n", "stop = timeit.default_timer()\n", "mongo_read_execution_time = stop - start" ] }, { "cell_type": "code", "execution_count": 73, "id": "93fb22ea-b283-4447-b774-fe755a782223", "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "56.66832709600567\n" ] } ], "source": [ "print(mongo_read_execution_time)" ] }, { "cell_type": "markdown", "id": "97405e42-61dc-42c7-8220-237a312c0ec7", "metadata": { "tags": [] }, "source": [ "### DuckDB" ] }, { "cell_type": "code", "execution_count": 85, "id": "bbcdb883-d6dc-46db-88db-4c90b84522ba", "metadata": {}, "outputs": [], "source": [ "# https://duckdb.org/2022/07/27/art-storage.html\n", "\n", "\n", "def duckdbConnect():\n", " cursor = duckdb.connect()\n", " return cursor\n", "\n", "\n", "def duckdbLoadCsv(csvFile=\"out.csv\"):\n", " data = pd.read_csv(\"out.csv\")\n", " return data\n", "\n", "\n", "# write\n", "def duckdbWrite(file, dataframe, tableName):\n", " conn = duckdbConnect()\n", " conn = duckdb.connect(file)\n", " # conn.execute(\"DROP TABLE EURUSDtest\")\n", " # conn.sql('CREATE TABLE EURUSDtest(i INTEGER)')\n", " conn.register(\"tempTable\", dataframe)\n", " conn.execute(\"CREATE TABLE {} AS SELECT * FROM tempTable\".format(tableName))\n", " conn.close()\n", " return 0\n", "\n", "\n", "def duckdbRead(dbfile):\n", " conn = duckdbConnect()\n", " conn = duckdb.connect(dbfile)\n", " conn.execute(\"SELECT * FROM {}\".format(dbname)).fetchdf()\n", " conn.close()\n", " return 0" ] }, { "cell_type": "code", "execution_count": 73, "id": "1c787f48-5640-4eb5-9456-be8f0a8211eb", "metadata": { "tags": [] }, "outputs": [], "source": [ "data = duckdbLoadCsv()" ] }, { "cell_type": "code", "execution_count": null, "id": "f07f03d0-021e-4dc3-bfa8-efc029a9797a", "metadata": { "tags": [] }, "outputs": [], "source": [ "conn = duckdbConnect()\n", "conn.execute(\"DROP TABLE EURUSD\")" ] }, { "cell_type": "code", "execution_count": 74, "id": "45620be8-34c6-4a3a-89f7-09337f665c90", "metadata": { "tags": [] }, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 74, "metadata": {}, "output_type": "execute_result" } ], "source": [ "conn.register(\"EURUSDtest\", data)" ] }, { "cell_type": "code", "execution_count": 84, "id": "9eb19431-fbad-43b4-84f7-0043e65de162", "metadata": { "tags": [] }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
name
0EURUSDtest
\n", "
" ], "text/plain": [ " name\n", "0 EURUSDtest" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "conn = duckdb.connect(\"file.db\")\n", "# conn.execute(\"DROP TABLE EURUSDtest\")\n", "# conn.sql(\"CREATE TABLE EURUSDtest(i INTEGER)\")\n", "display(conn.execute(\"SHOW TABLES\").df())" ] }, { "cell_type": "code", "execution_count": null, "id": "32037939-def0-48e7-9a47-46bcf1a5883c", "metadata": { "tags": [] }, "outputs": [], "source": [ "# conn = duckdbConnect()\n", "conn.execute(\"SELECT * FROM EURUSDtest\")\n", "conn.fetchall()" ] }, { "cell_type": "code", "execution_count": 82, "id": "c6f53d67-684b-4b34-a573-472986ee3e47", "metadata": { "tags": [] }, "outputs": [], "source": [ "start = timeit.default_timer()\n", "dfDuckdb = duckdbWrite(\"file.db\", data, dbname)\n", "stop = timeit.default_timer()\n", "duckdb_write_execution_time = stop - start" ] }, { "cell_type": "code", "execution_count": 87, "id": "102f363a-b35d-433c-8752-7acc85c27bdc", "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "1.7639581979965442\n" ] } ], "source": [ "print(duckdb_write_execution_time)" ] }, { "cell_type": "code", "execution_count": 86, "id": "f630fc1a-0d52-4e3a-9dfe-1ec60d188033", "metadata": { "tags": [] }, "outputs": [], "source": [ "start = timeit.default_timer()\n", "dfDuckdb = duckdbRead(\"file.db\")\n", "stop = timeit.default_timer()\n", "duckdb_read_execution_time = stop - start" ] }, { "cell_type": "code", "execution_count": 88, "id": "c6abdaaa-3ac2-425b-9208-d6cb79afe966", "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0.5125257010004134\n" ] } ], "source": [ "print(duckdb_read_execution_time)" ] }, { "cell_type": "markdown", "id": "4409cc89-ed14-4313-ac89-65b826038533", "metadata": { "tags": [] }, "source": [ "### Kdb+" ] }, { "cell_type": "code", "execution_count": 31, "id": "14f63810-1943-4e28-9bce-2148be6be02d", "metadata": {}, "outputs": [], "source": [ "# numpy version boolean fix\n", "np.bool = np.bool_" ] }, { "cell_type": "code", "execution_count": 25, "id": "bbd217e3-695f-4fa6-ae42-83db1dde8311", "metadata": {}, "outputs": [], "source": [ "# functions\n", "# cd ~ && q/l64/q -p 5001\n", "\n", "\n", "def kdbConnect():\n", " q = qconnection.QConnection(host=\"localhost\", port=5001)\n", " q.open()\n", " return q\n", "\n", "\n", "def kdbLoadCsv(file=\"out.csv\"):\n", " data = pd.read_csv(file)\n", " return data\n", "\n", "\n", "def kdbWrite():\n", " q = kdbConnect()\n", " data = kdbLoadCsv()\n", " q.sendSync(\"{t::x}\", data)\n", " q.sendSync(\"`:/home/sandman/q/tab1 set t\")\n", " q.close()\n", " return 0\n", "\n", "\n", "def kdbRead():\n", " q = kdbConnect()\n", " # df2 = q.sendSync(\"tab2: get `:/home/sandman/q/tab1\")\n", " # df2 = q.sendSync(\"tab2\")\n", " df = pd.DataFrame(q(\"t\")) # , pandas=True))\n", " # df3 = q.sendSync(\"select from t\")\n", " # ver todos esses loads\n", " q.close()\n", " return 0" ] }, { "cell_type": "code", "execution_count": 28, "id": "67f0c26e-44fb-40b0-a147-5d97bfbbded2", "metadata": {}, "outputs": [], "source": [ "# write\n", "start = timeit.default_timer()\n", "dfKdb = kdbWrite()\n", "stop = timeit.default_timer()\n", "kdb_write_execution_time = stop - start" ] }, { "cell_type": "code", "execution_count": 29, "id": "dcb200be-ffc9-4bcc-8554-5740fb420ab5", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2.8739770100000896\n" ] } ], "source": [ "# print write time\n", "print(kdb_write_execution_time)" ] }, { "cell_type": "code", "execution_count": 30, "id": "d4ce0203-b0c7-440b-a3ca-d7b2a7682474", "metadata": {}, "outputs": [], "source": [ "# read\n", "start = timeit.default_timer()\n", "dfKdb = kdbRead()\n", "stop = timeit.default_timer()\n", "kdb_read_execution_time = stop - start" ] }, { "cell_type": "code", "execution_count": null, "id": "1a16fd76-2158-40fe-9285-c53791f8ed51", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "4.153738381999574\n" ] } ], "source": [ "# print read time\n", "print(kdb_read_execution_time)" ] }, { "cell_type": "code", "execution_count": 46, "id": "3a09558c-73e6-4324-9fc5-782fcd0d12e5", "metadata": { "tags": [] }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Write TimeRead TimeTotal Time
Kdb+2.87 sec4.15 sec7.03 sec
r2fill150
r3fill140
\n", "
" ], "text/plain": [ " Write Time Read Time Total Time\n", "Kdb+ 2.87 sec 4.15 sec 7.03 sec\n", "r2 fill 15 0\n", "r3 fill 14 0" ] }, "execution_count": 46, "metadata": {}, "output_type": "execute_result" } ], "source": [ "s = \" sec\"\n", "data = [\n", " [\n", " \"{:.2f}\".format(kdb_write_execution_time) + s,\n", " \"{:.2f}\".format(kdb_read_execution_time) + s,\n", " \"{:.2f}\".format(kdb_write_execution_time + kdb_read_execution_time) + s,\n", " ],\n", " [\"fill\", 15, 0],\n", " [\"fill\", 14, 0],\n", "]\n", "\n", "index_labels=['Kdb+','r2','r3']\n", "# Create the pandas DataFrame\n", "df = pd.DataFrame(data, columns=[\"Write Time\", \"Read Time\", \"Total Time\"],index=index_labels)\n", "\n", "# print dataframe.\n", "df" ] }, { "cell_type": "markdown", "id": "7baf1fd1-2afd-41b5-a579-33f053e4ddfc", "metadata": {}, "source": [ "## Graph\n" ] }, { "cell_type": "code", "execution_count": 30, "id": "a9740731-6077-4bf1-bb65-b4c4225ac79b", "metadata": { "tags": [] }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjsAAAGwCAYAAABPSaTdAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA8hElEQVR4nO3deVSU9f///8cgMCCrkLIoKomZlktqGmniguGapqX2NsPyrS1ameVWaaYZZrmkWZbvt1sfy7LUfOvJJfdcEEEtyz1cToqWCwgqIrx+f/R1fpE7DA5e3m/nzDnO63rNa57XcDk8eM3rusZmjDECAACwKDdXFwAAAFCUCDsAAMDSCDsAAMDSCDsAAMDSCDsAAMDSCDsAAMDSCDsAAMDS3F1dQHGQl5enw4cPy8/PTzabzdXlAACA62CM0enTpxUeHi43tyvP3xB2JB0+fFgRERGuLgMAABTAoUOHVK5cuStuJ+xI8vPzk/TXi+Xv7+/iagAAwPXIyMhQRESE4/f4lRB2JMdHV/7+/oQdAABuMddagsICZQAAYGmEHQAAYGmEHQAAYGms2QEAwIlyc3OVk5Pj6jIswcPDQyVKlCj0OIQdAACcwBijtLQ0nTp1ytWlWEpgYKBCQ0MLdR08wg4AAE5wMeiUKVNGJUuW5CK1hWSM0ZkzZ3Ts2DFJUlhYWIHHIuwAAFBIubm5jqATHBzs6nIsw9vbW5J07NgxlSlTpsAfabFAGQCAQrq4RqdkyZIursR6Lr6mhVkHRdgBAMBJ+OjK+ZzxmhJ2AACApRF2AACApbFAGQCAIlRx0KKb9lz7R7W+ac9VWN27d9epU6c0f/78In8uZnYAALiNde/eXTabTTabTR4eHoqMjNSAAQN07tw5V5fmNMzsAABwm2vRooWmTZumnJwcJScnKz4+XjabTe+9956rS3MKZnYAALjN2e12hYaGKiIiQu3bt1dsbKyWLVsmScrLy1NCQoIiIyPl7e2tmjVr6ptvvnE8Njc3Vz169HBsr1Klij788MN84+fm5qpfv34KDAxUcHCwBgwYIGPMTds/ZnYAi7uZ6wWKq1tpHQPgatu3b9f69etVoUIFSVJCQoL+7//+T5MnT1blypW1Zs0aPfnkkypdurRiYmKUl5encuXKac6cOQoODtb69evVq1cvhYWFqVOnTpKkMWPGaPr06Zo6daqqVq2qMWPGaN68eWratOlN2SfCDgAAt7mFCxfK19dXFy5cUHZ2ttzc3PTRRx8pOztb7777rn744QdFR0dLku688079+OOP+vTTTxUTEyMPDw+9/fbbjrEiIyO1YcMGff31146wM378eA0ePFgdOnSQJE2ePFlLliy5aftH2AEA4DbXpEkTffLJJ8rKytK4cePk7u6ujh076pdfftGZM2fUvHnzfP3Pnz+v++67z3F/0qRJmjp1qg4ePKizZ8/q/PnzqlWrliQpPT1dR44cUf369R393d3dVbdu3Zv2URZhBwCA25yPj4+ioqIkSVOnTlXNmjX13//+V/fee68kadGiRSpbtmy+x9jtdknS7Nmz9dprr2nMmDGKjo6Wn5+f3n//fSUmJt7cnbgKwg4AAHBwc3PT66+/rn79+mn37t2y2+06ePCgYmJiLtt/3bp1evDBB/XCCy842vbt2+f4d0BAgMLCwpSYmKhGjRpJki5cuKDk5GTVrl27aHfm/yHsAACAfB5//HH1799fn376qV577TW98sorysvLU8OGDZWenq5169bJ399f8fHxqly5smbOnKklS5YoMjJSn3/+uZKSkhQZGekY7+WXX9aoUaNUuXJl3X333Ro7dqxOnTp10/aHsAMAQBG6Fc8GdHd3V58+fTR69GilpqaqdOnSSkhI0G+//abAwEDVrl1br7/+uiTp2Wef1ZYtW9S5c2fZbDY98cQTeuGFF/T99987xnv11Vd15MgRxcfHy83NTc8884weffRRpaen35T9sZmbeaJ7MZWRkaGAgAClp6fL39/f1eUATsWp57fmLxvcWs6dO6fU1FRFRkbKy8vL1eVYytVe2+v9/c1FBQEAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKW5NOysWbNGbdu2VXh4uGw2m+bPn+/YlpOTo4EDB6p69ery8fFReHi4nnrqKR0+fDjfGCdOnFDXrl3l7++vwMBA9ejRQ5mZmTd5TwAAQHHl0rCTlZWlmjVratKkSZdsO3PmjFJSUjRkyBClpKRo7ty52rVrlx555JF8/bp27apffvlFy5Yt08KFC7VmzRr16tXrZu0CAAAo5lx6BeWWLVuqZcuWl90WEBCgZcuW5Wv76KOPVK9ePR08eFDly5fXjh07tHjxYiUlJalu3bqSpIkTJ6pVq1b64IMPFB4eXuT7AAAApIoVK6pv377q27evq0u5xC31dRHp6emy2WwKDAyUJG3YsEGBgYGOoCNJsbGxcnNzU2Jioh599NHLjpOdna3s7GzH/YyMjCKtGwBwGxsWcBOf68a+fmHy5Mnq37+/Tp48KXf3vyJBZmamSpUqpQYNGmjVqlWOvqtWrVKTJk20d+9eVapU6ZKxkpKS5OPj47hvs9k0b948tW/fvkC74ky3zALlc+fOaeDAgXriiSccl4ROS0tTmTJl8vVzd3dXUFCQ0tLSrjhWQkKCAgICHLeIiIgirR0AgOKoSZMmyszM1ObNmx1ta9euVWhoqBITE3Xu3DlH+8qVK1W+fPlLgs758+clSaVLl1bJkiVvTuE36JYIOzk5OerUqZOMMfrkk08KPd7gwYOVnp7uuB06dMgJVQIAcGupUqWKwsLCLpnBadeunSIjI7Vx48Z87U2aNFH37t3Vvn17jRw5UuHh4apSpYqkvz7GGj9+vOPfkvToo4/KZrM57kvSd999p9q1a8vLy0t33nmn3n77bV24cKFI97PYh52LQefAgQNatmxZvi/6Cg0N1bFjx/L1v3Dhgk6cOKHQ0NArjmm32+Xv75/vBgDA7ahJkyZauXKl4/7KlSvVuHFjxcTEONrPnj2rxMRENWnSRJK0fPly7dq1y3Fy0D8lJSVJkqZNm6YjR4447q9du1ZPPfWUXn75Zf3666/69NNPNX36dI0cObJI97FYh52LQWfPnj364YcfFBwcnG97dHS0Tp06peTkZEfbihUrlJeXp/r169/scgEAuOU0adJE69at04ULF3T69Glt2bJFMTExatSokWPGZ8OGDcrOznaEHR8fH/3nP//RPffco3vuueeSMUuXLi1JCgwMVGhoqOP+22+/rUGDBik+Pl533nmnmjdvrhEjRujTTz8t0n106QLlzMxM7d2713E/NTVVW7duVVBQkMLCwvTYY48pJSVFCxcuVG5urmMdTlBQkDw9PVW1alW1aNFCPXv21OTJk5WTk6M+ffqoS5cunIkFAMB1aNy4sbKyspSUlKSTJ0/qrrvuUunSpRUTE6Onn35a586d06pVq3TnnXeqfPnykqTq1avL09Pzhp9r27ZtWrduXb6ZnNzcXJ07d05nzpwpsjU/Lg07mzdvdqRESerXr58kKT4+XsOGDdOCBQskSbVq1cr3uItTbJI0a9Ys9enTR82aNZObm5s6duyoCRMm3JT6AQC41UVFRalcuXJauXKlTp48qZiYGElSeHi4IiIitH79eq1cuVJNmzZ1PObvZ13diMzMTL399tvq0KHDJdu8vLwKtgPXwaVhp3HjxjLGXHH71bZdFBQUpC+++MKZZQEAcFtp0qSJVq1apZMnT6p///6O9kaNGun777/Xpk2b9Pzzz9/QmB4eHsrNzc3XVrt2be3atUtRUVFOqft63VLX2QEAAM7XpEkT9e7dWzk5OY6ZHUmKiYlRnz59dP78+XyfxFyPihUravny5WrQoIHsdrtKlSqloUOHqk2bNipfvrwee+wxubm5adu2bdq+fbveeecdZ++WQ7FeoAwAAIpekyZNdPbsWUVFRSkkJMTRHhMTo9OnTztOUb8RY8aM0bJlyxQREaH77rtPkhQXF6eFCxdq6dKluv/++/XAAw9o3LhxqlChglP3559s5no+K7K4jIwMBQQEKD09ndPQYTkVBy1ydQkut39Ua1eXAIs7d+6cUlNTFRkZWaRrT25HV3ttr/f3NzM7AADA0gg7AADA0gg7AADA0gg7AADA0gg7AAA4Cef8OJ8zXlPCDgAAheTh4SFJOnPmjIsrsZ6Lr+nF17gguKggAACFVKJECQUGBurYsWOSpJIlS8pms7m4qlubMUZnzpzRsWPHFBgYqBIlShR4LMIOAABOEBoaKkmOwAPnuPjN6YVB2AEAwAlsNpvCwsJUpkwZ5eTkuLocS/Dw8CjUjM5FhB0AAJyoRIkSTvkFDedhgTIAALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0l4adNWvWqG3btgoPD5fNZtP8+fPzbTfGaOjQoQoLC5O3t7diY2O1Z8+efH1OnDihrl27yt/fX4GBgerRo4cyMzNv4l4AAIDizKVhJysrSzVr1tSkSZMuu3306NGaMGGCJk+erMTERPn4+CguLk7nzp1z9Onatat++eUXLVu2TAsXLtSaNWvUq1evm7ULAACgmHN35ZO3bNlSLVu2vOw2Y4zGjx+vN998U+3atZMkzZw5UyEhIZo/f766dOmiHTt2aPHixUpKSlLdunUlSRMnTlSrVq30wQcfKDw8/LJjZ2dnKzs723E/IyPDyXsGAACKi2K7Zic1NVVpaWmKjY11tAUEBKh+/frasGGDJGnDhg0KDAx0BB1Jio2NlZubmxITE684dkJCggICAhy3iIiIotsRAADgUsU27KSlpUmSQkJC8rWHhIQ4tqWlpalMmTL5tru7uysoKMjR53IGDx6s9PR0x+3QoUNOrh4AABQXLv0Yy1XsdrvsdrurywAAADdBsZ3ZCQ0NlSQdPXo0X/vRo0cd20JDQ3Xs2LF82y9cuKATJ044+gAAgNtbsQ07kZGRCg0N1fLlyx1tGRkZSkxMVHR0tCQpOjpap06dUnJysqPPihUrlJeXp/r169/0mgEAQPHj0o+xMjMztXfvXsf91NRUbd26VUFBQSpfvrz69u2rd955R5UrV1ZkZKSGDBmi8PBwtW/fXpJUtWpVtWjRQj179tTkyZOVk5OjPn36qEuXLlc8EwsAANxeXBp2Nm/erCZNmjju9+vXT5IUHx+v6dOna8CAAcrKylKvXr106tQpNWzYUIsXL5aXl5fjMbNmzVKfPn3UrFkzubm5qWPHjpowYcJN3xcAAFA82YwxxtVFuFpGRoYCAgKUnp4uf39/V5cDOFXFQYtcXYLL7R/V2tUlACgC1/v7u9iu2QEAAHAGwg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALC0Yh12cnNzNWTIEEVGRsrb21uVKlXSiBEjZIxx9DHGaOjQoQoLC5O3t7diY2O1Z88eF1YNAACKk2Iddt577z198skn+uijj7Rjxw699957Gj16tCZOnOjoM3r0aE2YMEGTJ09WYmKifHx8FBcXp3PnzrmwcgAAUFy4u7qAq1m/fr3atWun1q1bS5IqVqyoL7/8Ups2bZL016zO+PHj9eabb6pdu3aSpJkzZyokJETz589Xly5dXFY7AAAoHor1zM6DDz6o5cuXa/fu3ZKkbdu26ccff1TLli0lSampqUpLS1NsbKzjMQEBAapfv742bNhwxXGzs7OVkZGR7wYAAKypWM/sDBo0SBkZGbr77rtVokQJ5ebmauTIkerataskKS0tTZIUEhKS73EhISGObZeTkJCgt99+u+gKBwAAxUaxntn5+uuvNWvWLH3xxRdKSUnRjBkz9MEHH2jGjBmFGnfw4MFKT0933A4dOuSkigEAQHFTrGd2+vfvr0GDBjnW3lSvXl0HDhxQQkKC4uPjFRoaKkk6evSowsLCHI87evSoatWqdcVx7Xa77HZ7kdYOAACKh2I9s3PmzBm5ueUvsUSJEsrLy5MkRUZGKjQ0VMuXL3dsz8jIUGJioqKjo29qrQAAoHgq1jM7bdu21ciRI1W+fHndc8892rJli8aOHatnnnlGkmSz2dS3b1+98847qly5siIjIzVkyBCFh4erffv2ri0eAAAUC8U67EycOFFDhgzRCy+8oGPHjik8PFzPPvushg4d6ugzYMAAZWVlqVevXjp16pQaNmyoxYsXy8vLy4WVAwCA4sJm/n454ttURkaGAgIClJ6eLn9/f1eXAzhVxUGLXF2Cy+0f1drVJQAoAtf7+7tYr9kBAAAoLMIOAACwNMIOAACwNMIOAACwNMIOAACwtAKFnbNnz+rMmTOO+wcOHND48eO1dOlSpxUGAADgDAUKO+3atdPMmTMlSadOnVL9+vU1ZswYtWvXTp988olTCwQAACiMAoWdlJQUPfTQQ5Kkb775RiEhITpw4IBmzpypCRMmOLVAAACAwihQ2Dlz5oz8/PwkSUuXLlWHDh3k5uamBx54QAcOHHBqgQAAAIVRoLATFRWl+fPn69ChQ1qyZIkefvhhSdKxY8e4AjEAAChWChR2hg4dqtdee00VK1ZU/fr1Hd8wvnTpUt13331OLRAAAKAwCvRFoI899pgaNmyoI0eOqGbNmo72Zs2a6dFHH3VacQAAAIVV4G89Dw0NVWhoaL62evXqFbogAAAAZ7rusNOhQ4frHnTu3LkFKgYAAMDZrnvNTkBAgOPm7++v5cuXa/PmzY7tycnJWr58uQICAoqkUAAAgIK47pmdadOmOf49cOBAderUSZMnT1aJEiUkSbm5uXrhhRc4GwsAABQrBToba+rUqXrttdccQUeSSpQooX79+mnq1KlOKw4AAKCwChR2Lly4oJ07d17SvnPnTuXl5RW6KAAAAGcp0NlYTz/9tHr06KF9+/Y5zsBKTEzUqFGj9PTTTzu1QAAAgMIoUNj54IMPFBoaqjFjxujIkSOSpLCwMPXv31+vvvqqUwsEAAAojAKFHTc3Nw0YMEADBgxQRkaGJLEwGQAAFEsFvqjgRYQcAABQnBVogfLRo0fVrVs3hYeHy93dXSVKlMh3AwAAKC4KNLPTvXt3HTx4UEOGDFFYWJhsNpuz6wIAAHCKAoWdH3/8UWvXrlWtWrWcXA4AAIBzFehjrIiICBljnF0LAACA0xUo7IwfP16DBg3S/v37nVwOAACAcxXoY6zOnTvrzJkzqlSpkkqWLCkPD49820+cOOGU4gAAAAqrQGFn/PjxTi4DAACgaBQo7MTHxzu7DgAAgCJR4IsK5ubmav78+dqxY4ck6Z577tEjjzzCdXYAAECxUqCws3fvXrVq1Uq///67qlSpIklKSEhQRESEFi1apEqVKjm1SAAAgIIq0NlYL730kipVqqRDhw4pJSVFKSkpOnjwoCIjI/XSSy85u0YAAIACK9DMzurVq7Vx40YFBQU52oKDgzVq1Cg1aNDAacUBAAAUVoFmdux2u06fPn1Je2Zmpjw9PQtdFAAAgLMUKOy0adNGvXr1UmJioowxMsZo48aNeu655/TII484u0YAAIACK1DYmTBhgipVqqTo6Gh5eXnJy8tLDRo0UFRUlD788ENn1wgAAFBgBVqzExgYqO+++0579+51nHpetWpVRUVFObU4AACAwirwdXYkKSoqioADAACKtQJ9jNWxY0e99957l7SPHj1ajz/+eKGLAgAAcJYChZ01a9aoVatWl7S3bNlSa9asKXRRAAAAzlKgsHOlU8w9PDyUkZFR6KIAAACcpUBhp3r16vrqq68uaZ89e7aqVatW6KIAAACcpUALlIcMGaIOHTpo3759atq0qSRp+fLl+vLLLzVnzhynFggAAFAYBQo7bdu21fz58/Xuu+/qm2++kbe3t2rUqKEffvhBMTExzq4RAACgwAr0MZYktW7dWuvWrVNWVpb+/PNPrVixokiCzu+//64nn3xSwcHB8vb2VvXq1bV582bHdmOMhg4dqrCwMHl7eys2NlZ79uxxeh0AAODWVOCwc+rUKf3nP//R66+/rhMnTkiSUlJS9PvvvzutuJMnT6pBgwby8PDQ999/r19//VVjxoxRqVKlHH1Gjx6tCRMmaPLkyUpMTJSPj4/i4uJ07tw5p9UBAABuXQX6GOunn35SbGysAgICtH//fv373/9WUFCQ5s6dq4MHD2rmzJlOKe69995TRESEpk2b5miLjIx0/NsYo/Hjx+vNN99Uu3btJEkzZ85USEiI5s+fry5dujilDgAAcOsq0MxOv3791L17d+3Zs0deXl6O9latWjn1OjsLFixQ3bp19fjjj6tMmTK67777NGXKFMf21NRUpaWlKTY21tEWEBCg+vXra8OGDVccNzs7WxkZGfluAADAmgoUdpKSkvTss89e0l62bFmlpaUVuqiLfvvtN33yySeqXLmylixZoueff14vvfSSZsyYIUmO5woJCcn3uJCQkKvWkZCQoICAAMctIiLCaTUDAIDipUBhx263X3Y2ZPfu3SpdunShi7ooLy9PtWvX1rvvvqv77rtPvXr1Us+ePTV58uRCjTt48GClp6c7bocOHXJSxQAAoLgpUNh55JFHNHz4cOXk5EiSbDabDh48qIEDB6pjx45OKy4sLOySixRWrVpVBw8elCSFhoZKko4ePZqvz9GjRx3bLsdut8vf3z/fDQAAWFOBws6YMWOUmZmpMmXK6OzZs4qJiVGlSpXk6+urkSNHOq24Bg0aaNeuXfnadu/erQoVKkj6a7FyaGioli9f7tiekZGhxMRERUdHO60OAABw6yrQ2VgBAQFatmyZfvzxR/3000/KzMxUnTp11KxZM6cW98orr+jBBx/Uu+++q06dOmnTpk367LPP9Nlnn0n6a0apb9++euedd1S5cmVFRkZqyJAhCg8PV/v27Z1aCwAAuDXdUNjZsGGDjh8/rjZt2kiSGjZsqH379mn06NE6c+aM2rdvr4kTJ8putzuluPvvv1/z5s3T4MGDNXz4cEVGRmr8+PHq2rWro8+AAQOUlZWlXr166dSpU2rYsKEWL16c7ywxAABw+7IZY8z1dm7ZsqUaN26sgQMHSpJ+/vln1alTR/Hx8apataref/99Pfvssxo2bFhR1VskMjIyFBAQoPT0dNbvwHIqDlrk6hJcbv+o1q4uAUARuN7f3ze0Zmfr1q35PqqaPXu26tWrpylTpqhfv36aMGGCvv7664JXDQAA4GQ3FHZOnjyZ75o2q1evVsuWLR3377//fk7jBgAAxcoNhZ2QkBClpqZKks6fP6+UlBQ98MADju2nT5+Wh4eHcysEAAAohBsKO61atdKgQYO0du1aDR48WCVLltRDDz3k2P7TTz+pUqVKTi8SAACgoG7obKwRI0aoQ4cOiomJka+vr2bMmCFPT0/H9qlTp+rhhx92epEAAAAFdUNh54477tCaNWuUnp4uX19flShRIt/2OXPmyNfX16kFAgAAFEaBLyp4OUFBQYUqBgAAwNkK9HURAAAAtwrCDgAAsDTCDgAAsDTCDgAAsDTCDgAAsDTCDgAAsDTCDgAAsDTCDgAAsDTCDgAAsDTCDgAAsDTCDgAAsDTCDgAAsDTCDgAAsDTCDgAAsDTCDgAAsDTCDgAAsDTCDgAAsDTCDgAAsDTCDgAAsDTCDgAAsDTCDgAAsDTCDgAAsDTCDgAAsDTCDgAAsDTCDgAAsDTCDgAAsDTCDgAAsDTCDgAAsDTCDgAAsDTCDgAAsDTCDgAAsDTCDgAAsDTCDgAAsDTCDgAAsDTCDgAAsDTCDgAAsDTCDgAAsDTCDgAAsLRbKuyMGjVKNptNffv2dbSdO3dOvXv3VnBwsHx9fdWxY0cdPXrUdUUCAIBi5ZYJO0lJSfr0009Vo0aNfO2vvPKK/ve//2nOnDlavXq1Dh8+rA4dOrioSgAAUNzcEmEnMzNTXbt21ZQpU1SqVClHe3p6uv773/9q7Nixatq0qerUqaNp06Zp/fr12rhx4xXHy87OVkZGRr4bAACwplsi7PTu3VutW7dWbGxsvvbk5GTl5OTka7/77rtVvnx5bdiw4YrjJSQkKCAgwHGLiIgostoBAIBrFfuwM3v2bKWkpCghIeGSbWlpafL09FRgYGC+9pCQEKWlpV1xzMGDBys9Pd1xO3TokLPLBgAAxYS7qwu4mkOHDunll1/WsmXL5OXl5bRx7Xa77Ha708YDAADFV7Ge2UlOTtaxY8dUu3Ztubu7y93dXatXr9aECRPk7u6ukJAQnT9/XqdOncr3uKNHjyo0NNQ1RQMAgGKlWM/sNGvWTD///HO+tqefflp33323Bg4cqIiICHl4eGj58uXq2LGjJGnXrl06ePCgoqOjXVEyAAAoZop12PHz89O9996br83Hx0fBwcGO9h49eqhfv34KCgqSv7+/XnzxRUVHR+uBBx5wRckAAKCYKdZh53qMGzdObm5u6tixo7KzsxUXF6ePP/7Y1WUBAIBiwmaMMa4uwtUyMjIUEBCg9PR0+fv7u7ocwKkqDlrk6hJcbv+o1q4uAUARuN7f38V6gTIAAEBhEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClubu6AAAocsMCXF2Baw1Ld3UFgEsxswMAACyNsAMAACyNsAMAACyNsAMAACyNsAMAACyNsAMAACyNsAMAACyNsAMAACyNsAMAACyNsAMAACyNsAMAACyNsAMAACyNsAMAACyNsAMAACytWIedhIQE3X///fLz81OZMmXUvn177dq1K1+fc+fOqXfv3goODpavr686duyoo0ePuqhiAABQ3BTrsLN69Wr17t1bGzdu1LJly5STk6OHH35YWVlZjj6vvPKK/ve//2nOnDlavXq1Dh8+rA4dOriwagAAUJy4u7qAq1m8eHG++9OnT1eZMmWUnJysRo0aKT09Xf/973/1xRdfqGnTppKkadOmqWrVqtq4caMeeOABV5QNAACKkWI9s/NP6enpkqSgoCBJUnJysnJychQbG+voc/fdd6t8+fLasGHDFcfJzs5WRkZGvhsAALCmWybs5OXlqW/fvmrQoIHuvfdeSVJaWpo8PT0VGBiYr29ISIjS0tKuOFZCQoICAgIct4iIiKIsHQAAuNAtE3Z69+6t7du3a/bs2YUea/DgwUpPT3fcDh065IQKAQBAcVSs1+xc1KdPHy1cuFBr1qxRuXLlHO2hoaE6f/68Tp06lW925+jRowoNDb3ieHa7XXa7vShLBgAAxUSxntkxxqhPnz6aN2+eVqxYocjIyHzb69SpIw8PDy1fvtzRtmvXLh08eFDR0dE3u1wAAFAMFeuZnd69e+uLL77Qd999Jz8/P8c6nICAAHl7eysgIEA9evRQv379FBQUJH9/f7344ouKjo7mTCwAACCpmIedTz75RJLUuHHjfO3Tpk1T9+7dJUnjxo2Tm5ubOnbsqOzsbMXFxenjjz++yZUCAIDiqliHHWPMNft4eXlp0qRJmjRp0k2oCAAA3GqK9ZodAACAwiLsAAAASyPsAAAASyvWa3asoOKgRa4uweX2j2rt6hIAALcxZnYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClubu6ANwGhgW4ugLXGpbu6goA4LbGzA4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0Tj0HAKCocQkOlz49MzsAAMDSCDsAAMDSCDsAAMDSCDsAAMDSCDsAAMDSCDsAAMDSLBN2Jk2apIoVK8rLy0v169fXpk2bXF0SAAAoBiwRdr766iv169dPb731llJSUlSzZk3FxcXp2LFjri4NAAC4mCUuKjh27Fj17NlTTz/9tCRp8uTJWrRokaZOnapBgwa5uDoAuL1VHLTI1SW43H4vV1dwe7vlw8758+eVnJyswYMHO9rc3NwUGxurDRs2XPYx2dnZys7OdtxPT//ryo4ZGRlOry8v+4zTx7zVZNiMq0twrSI4rm4ExyDHIMeg63EMFs0xePH3tjFXf31v+bDz559/Kjc3VyEhIfnaQ0JCtHPnzss+JiEhQW+//fYl7REREUVS4+3uNr9IujTqtn8FXO62/wlwDLrcbf8TKOJj8PTp0woIuPJz3PJhpyAGDx6sfv36Oe7n5eXpxIkTCg4Ols1mc2Fl1pORkaGIiAgdOnRI/v7+ri4HtyGOQbgax2DRMcbo9OnTCg8Pv2q/Wz7s3HHHHSpRooSOHj2ar/3o0aMKDQ297GPsdrvsdnu+tsDAwKIqEZL8/f35Tw6X4hiEq3EMFo2rzehcdMufjeXp6ak6depo+fLljra8vDwtX75c0dHRLqwMAAAUB7f8zI4k9evXT/Hx8apbt67q1aun8ePHKysry3F2FgAAuH1ZIux07txZf/zxh4YOHaq0tDTVqlVLixcvvmTRMm4+u92ut95665KPDYGbhWMQrsYx6Ho2c63ztQAAAG5ht/yaHQAAgKsh7AAAAEsj7AAAAEsj7NxGbDab5s+fL0nav3+/bDabtm7del2P7d69u9q3b19ktQFWVrFiRY0fP97VZQC3LcKORaSlpenFF1/UnXfeKbvdroiICLVt2zbf9Yf+LiIiQkeOHNG9995bZDX9PVz9HcHp9tC9e3fZbDbZbDZ5enoqKipKw4cP14ULFwo17qpVq2Sz2XTq1CnnFArL+eOPP/T888+rfPnystvtCg0NVVxcnNatW+fo8+yzz6pSpUry9vZW6dKl1a5duyt+xdBFjRs3dhzTXl5eqlatmj7++OOi3h2nuZ1DN2HHAvbv3686depoxYoVev/99/Xzzz9r8eLFatKkiXr37n3Zx5QoUUKhoaFyd7fE1QdQTLVo0UJHjhzRnj179Oqrr2rYsGF6//33XV2Ww/nz511dAopAx44dtWXLFs2YMUO7d+/WggUL1LhxYx0/ftzRp06dOpo2bZp27NihJUuWyBijhx9+WLm5uVcdu2fPnjpy5Ih+/fVXderUSb1799aXX35ZoDo5/m4ig1tey5YtTdmyZU1mZuYl206ePOn4tyQzb948Y4wxqampRpLZsmWLY/v27dtN69atjZ+fn/H19TUNGzY0e/fuNcYYEx8fb9q1a+fou2nTJnPHHXeYUaNGXbGuvz/f3/1zrHPnzpkXX3zRlC5d2tjtdtOgQQOzadMmx/Zp06aZgICAfGPMmzfP/P3w3bp1q2ncuLHx9fU1fn5+pnbt2iYpKcmxfe3ataZhw4bGy8vLlCtXzrz44ouXfb3gPP/8ORtjTPPmzc0DDzxgTpw4Ybp162YCAwONt7e3adGihdm9e7ej3/79+02bNm1MYGCgKVmypKlWrZpZtGiR47j9+y0+Pt4YY0xGRob517/+ZUqWLGlCQ0PN2LFjTUxMjHn55Zcd41aoUMEMHz7cdOvWzfj5+Tkee63jY9KkSSYqKsrY7XZTpkwZ07FjR8e2zMxM061bN+Pj42NCQ0PNBx98cNnnHTdunLNeWlzFyZMnjSSzatWqG3rctm3bjCTHe97l/PPnaowxlStXNl26dDHGGDNgwABTuXJl4+3tbSIjI82bb75pzp8/7+j71ltvmZo1a5opU6aYihUrGpvNZowxZvfu3eahhx4ydrvdVK1a1SxdujTf++fKlSuNpHzv51u2bDGSTGpqqqPtasdxTEzMJf93bifM7NziTpw4ocWLF6t3797y8fG5ZPv1fufX77//rkaNGslut2vFihVKTk7WM888c9mPHFasWKHmzZtr5MiRGjhwYGF3QQMGDNC3336rGTNmKCUlRVFRUYqLi9OJEyeue4yuXbuqXLlySkpKUnJysgYNGiQPDw9J0r59+9SiRQt17NhRP/30k7766iv9+OOP6tOnT6Frx43x9vbW+fPn1b17d23evFkLFizQhg0bZIxRq1atlJOTI0nq3bu3srOztWbNGv38889677335Ovrq4iICH377beSpF27dunIkSP68MMPJf11JfV169ZpwYIFWrZsmdauXauUlJRLavjggw9Us2ZNbdmyRUOGDLnm8bF582a99NJLGj58uHbt2qXFixerUaNGjvH69++v1atX67vvvtPSpUu1atWqyz4vbg5fX1/5+vpq/vz5ys7Ovq7HZGVladq0aYqMjFRERMQNPd/FY1qS/Pz8NH36dP3666/68MMPNWXKFI0bNy5f/7179+rbb7/V3LlztXXrVuXl5alDhw7y9PRUYmKiJk+eXKD31Wsdx3PnzlW5cuU0fPhwHTlyREeOHLnh57iluTptoXASExONJDN37txr9tVVZnYGDx5sIiMj8/0V8ncX/0qfO3eu8fX1NbNnz76u5/Py8jI+Pj75bu7u7o6/+DMzM42Hh4eZNWuW43Hnz5834eHhZvTo0caY65vZ8fPzM9OnT79sHT169DC9evXK17Z27Vrj5uZmzp49e839QMH8fWYnLy/PLFu2zNjtdtO+fXsjyaxbt87R988//zTe3t7m66+/NsYYU716dTNs2LDLjnu5v3IzMjKMh4eHmTNnjqPt1KlTpmTJkpfMsLRv3z7feNc6Pr799lvj7+9vMjIyLqnl9OnTxtPT01G3McYcP37ceHt7M7PjQt98840pVaqU8fLyMg8++KAZPHiw2bZt2yX9Jk2aZHx8fIwkU6VKlavO6hiTf2bnwoUL5vPPPzeSzEcffXTZ/u+//76pU6eO4/5bb71lPDw8zLFjxxxtS5YsMe7u7ub33393tH3//fc3PLNzPe9zt/NxyMzOLc446QLYW7du1UMPPeSYDbmcxMREPf744/r888/VuXPn6xp33Lhx2rp1a77bI4884ti+b98+5eTkqEGDBo42Dw8P1atXTzt27Lju+vv166d///vfio2N1ahRo7Rv3z7Htm3btmn69OmOv/h8fX0VFxenvLw8paamXvdz4MYtXLhQvr6+8vLyUsuWLdW5c2d1795d7u7uql+/vqNfcHCwqlSp4viZv/TSS3rnnXfUoEEDvfXWW/rpp5+u+jy//fabcnJyVK9ePUdbQECAqlSpcknfunXr5rt/reOjefPmqlChgu68805169ZNs2bN0pkzZyT9dfyeP38+374EBQVd9nlx83Ts2FGHDx/WggUL1KJFC61atUq1a9fW9OnT8/Xr2rWrtmzZotWrV+uuu+5Sp06ddO7cuauO/fHHH8vX11fe3t7q2bOnXnnlFT3//POSpK+++koNGjRQaGiofH199eabb+rgwYP5Hl+hQgWVLl3acX/Hjh2KiIhQeHi4o60gX2LN+9zVEXZucZUrV5bNZrvmWQTX4u3tfc0+lSpV0t13362pU6c6Pm64ltDQUEVFReW7+fn53VBtbm5ul4S6fz7/sGHD9Msvv6h169ZasWKFqlWrpnnz5kmSMjMz9eyzz+YLXNu2bdOePXtUqVKlG6oFN6ZJkybaunWr9uzZo7Nnz2rGjBmy2WzXfNy///1v/fbbb+rWrZt+/vln1a1bVxMnTnRKTf/8uPdax4efn59SUlL05ZdfKiwsTEOHDlXNmjU5G6yY8/LyUvPmzTVkyBCtX79e3bt311tvvZWvT0BAgCpXrqxGjRrpm2++0c6dOx3vG1fStWtXbd26VampqcrKytLYsWPl5uamDRs2qGvXrmrVqpUWLlyoLVu26I033rhkEfLllhtci5vbX7+q//4++M/3QN7nro6wc4sLCgpSXFycJk2apKysrEu2X+8bco0aNbR27dqrhpg77rhDK1as0N69e9WpU6frDjxXU6lSJXl6euY7JTQnJ0dJSUmqVq2aJKl06dI6ffp0vv273PWB7rrrLr3yyitaunSpOnTooGnTpkmSateurV9//fWS0BUVFSVPT89C7wOuzMfHR1FRUSpfvrzjzL+qVavqwoULSkxMdPQ7fvy4du3a5fiZS39dHuG5557T3Llz9eqrr2rKlCmS5PiZ/f2smTvvvFMeHh5KSkpytKWnp2v37t3XrPF6jg93d3fFxsZq9OjR+umnn7R//36tWLFClSpVkoeHR759OXny5HU9L26uatWqXfY98iJjjIwx11znExAQoKioKJUtW9YRQiRp/fr1qlChgt544w3VrVtXlStX1oEDB65ZV9WqVXXo0KF8a2g2btyYr8/FmaC/9/nne+D1HMeenp7XPNvMqgg7FjBp0iTl5uaqXr16+vbbb7Vnzx7t2LFDEyZMuO7p0D59+igjI0NdunTR5s2btWfPHn3++efatWtXvn5lypTRihUrtHPnTj3xxBOFvmaKj4+Pnn/+efXv31+LFy/Wr7/+qp49e+rMmTPq0aOHJKl+/foqWbKkXn/9de3bt09ffPFFvunos2fPqk+fPlq1apUOHDigdevWKSkpSVWrVpUkDRw4UOvXr1efPn0cswzfffcdC5RdpHLlymrXrp169uypH3/8Udu2bdOTTz6psmXLql27dpKkvn37asmSJUpNTVVKSopWrlzp+HlWqFBBNptNCxcu1B9//KHMzEz5+fkpPj5e/fv318qVK/XLL7+oR48ecnNzu+ZM0rWOj4ULF2rChAnaunWrDhw4oJkzZyovL09VqlSRr6+vevToof79+2vFihXavn27unfvnu+XIG6u48ePq2nTpvq///s//fTTT0pNTdWcOXM0evRox/H122+/KSEhQcnJyTp48KDWr1+vxx9/XN7e3mrVqlWBnrdy5co6ePCgZs+erX379mnChAnXnCWSpNjYWN11112Kj4/Xtm3btHbtWr3xxhv5+kRFRSkiIkLDhg3Tnj17tGjRIo0ZMyZfn+t5n6tYsaLWrFmj33//XX/++WeB9vOW5dIVQ3Caw4cPm969e5sKFSoYT09PU7ZsWfPII4+YlStXOvroGqeeb9u2zTz88MOmZMmSxs/Pzzz00ENm3759xphLTyM+fPiwueuuu0ynTp3MhQsXLlvT35/v7/451tmzZ82LL75o7rjjjsueem7MXwuSo6KijLe3t2nTpo357LPPHAuUs7OzTZcuXUxERITx9PQ04eHhpk+fPvkWH2/atMk0b97c+Pr6Gh8fH1OjRg0zcuTI63hlUVCXO/X8oounngcEBBhvb28TFxeX79TzPn36mEqVKhm73W5Kly5tunXrZv7880/H9uHDh5vQ0FBjs9mueup5vXr1zKBBgxyPu9ICzasdH2vXrjUxMTGmVKlSxtvb29SoUcN89dVXjseePn3aPPnkk6ZkyZImJCTEjB49mlPPXejcuXNm0KBBpnbt2iYgIMCULFnSVKlSxbz55pvmzJkzxhhjfv/9d9OyZUtTpkwZ4+HhYcqVK2f+9a9/mZ07d1517Mudev53/fv3N8HBwcbX19d07tzZjBs3Lt/JFRdPPf+nXbt2mYYNGxpPT09z1113mcWLF1/y/vnjjz+a6tWrGy8vL/PQQw+ZOXPmXHLq+bXe5zZs2GBq1Khh7Hb7bXfquc0YJ61wBYBiJCsrS2XLltWYMWMcs4Q3S+PGjVWrVq3b9mq1KDybzaZ58+ZxtXkn4fK5ACxhy5Yt2rlzp+rVq6f09HQNHz5ckhwfXQC4fRF2AFjGBx98oF27dsnT01N16tTR2rVrdccdd7i6LAAuxsdYAADA0jhlAAAAWBphBwAAWBphBwAAWBphBwAAWBphBwAAWBphB4DlrFq1SjabjS/rBCCJsAOgCHTv3l02m002m00eHh4KCQlR8+bNNXXqVOXl5V33ONOnT1dgYGDRFQrgtkDYAVAkWrRooSNHjmj//v36/vvv1aRJE7388stq06ZNob9AFgBuBGEHQJGw2+0KDQ1V2bJlVbt2bb3++uv67rvv9P333zu+tX7s2LGqXr26fHx8FBERoRdeeEGZmZmS/voo6umnn1Z6erpjlmjYsGGSpM8//1x169aVn5+fQkND9a9//UvHjh27pIZ169apRo0a8vLy0gMPPKDt27c7th0/flxPPPGEypYtq5IlS6p69er68ssv8z3+m2++UfXq1eXt7a3g4GDFxsYqKyvLsf0///mPqlatKi8vL9199936+OOPHdvOnz+vPn36KCwsTF5eXqpQoYISEhKc9fICuAGEHQA3TdOmTVWzZk3NnTtXkuTm5qYJEybol19+0YwZM7RixQoNGDBAkvTggw9q/Pjx8vf315EjR3TkyBG99tprkqScnByNGDFC27Zt0/z587V//3517979kufr37+/xowZo6SkJJUuXVpt27ZVTk6OJOncuXOqU6eOFi1apO3bt6tXr17q1q2bNm3aJEk6cuSInnjiCT3zzDPasWOHVq1apQ4dOujiRednzZqloUOHauTIkdqxY4feffddDRkyRDNmzJAkTZgwQQsWLNDXX3+tXbt2adasWapYsWJRvrwArsSl37kOwJLi4+NNu3btLrutc+fOpmrVqpfdNmfOHBMcHOy4P23aNBMQEHDN50tKSjKSzOnTp40xxqxcudJIMrNnz3b0OX78uPH29jZfffXVFcdp3bq1efXVV40xxiQnJxtJZv/+/ZftW6lSJfPFF1/kaxsxYoSJjo42xhjz4osvmqZNm5q8vLxr1g+gaDGzA+CmMsbIZrNJkn744Qc1a9ZMZcuWlZ+fn7p166bjx4/rzJkzVx0jOTlZbdu2Vfny5eXn56eYmBhJ0sGDB/P1i46Odvw7KChIVapU0Y4dOyRJubm5GjFihKpXr66goCD5+vpqyZIljjFq1qypZs2aqXr16nr88cc1ZcoUnTx5UpKUlZWlffv2qUePHvL19XXc3nnnHe3bt0/SX4u0t27dqipVquill17S0qVLnfDqASgIwg6Am2rHjh2KjIzU/v371aZNG9WoUUPffvutkpOTNWnSJEl/rXe5kqysLMXFxcnf31+zZs1SUlKS5s2bd83H/dP777+vDz/8UAMHDtTKlSu1detWxcXFOcYoUaKEli1bpu+//17VqlXTxIkTVaVKFaWmpjrWFU2ZMkVbt2513LZv366NGzdKkmrXrq3U1FSNGDFCZ8+eVadOnfTYY48V6DUDUDjuri4AwO1jxYoV+vnnn/XKK68oOTlZeXl5GjNmjNzc/vq76+uvv87X39PTU7m5ufnadu7cqePHj2vUqFGKiIiQJG3evPmyz7dx40aVL19eknTy5Ent3r1bVatWlfTX4uV27drpySeflCTl5eVp9+7dqlatmuPxNptNDRo0UIMGDTR06FBVqFBB8+bNU79+/RQeHq7ffvtNXbt2veL++vv7q3PnzurcubMee+wxtWjRQidOnFBQUNCNvGwAComwA6BIZGdnKy0tTbm5uTp69KgWL16shIQEtWnTRk899ZS2b9+unJwcTZw4UW3bttW6des0efLkfGNUrFhRmZmZWr58uWrWrKmSJUuqfPny8vT01MSJE/Xcc89p+/btGjFixGVrGD58uIKDgxUSEqI33nhDd9xxh9q3by9Jqly5sr755hutX79epUqV0tixY3X06FFH2ElMTNTy5cv18MMPq0yZMkpMTNQff/zhCEtvv/22XnrpJQUEBKhFixbKzs7W5s2bdfLkSfXr109jx45VWFiY7rvvPrm5uWnOnDkKDQ3lukGAK7h60RAA64mPjzeSjCTj7u5uSpcubWJjY83UqVNNbm6uo9/YsWNNWFiY8fb2NnFxcWbmzJlGkjl58qSjz3PPPWeCg4ONJPPWW28ZY4z54osvTMWKFY3dbjfR0dFmwYIFRpLZsmWLMeb/X6D8v//9z9xzzz3G09PT1KtXz2zbts0x7vHjx027du2Mr6+vKVOmjHnzzTfNU0895VhY/euvv5q4uDhTunRpY7fbzV133WUmTpyYbz9nzZplatWqZTw9PU2pUqVMo0aNzNy5c40xxnz22WemVq1axsfHx/j7+5tmzZqZlJQU57/YAK7JZsz/O48SAADAgligDAAALI2wAwAALI2wAwAALI2wAwAALI2wAwAALI2wAwAALI2wAwAALI2wAwAALI2wAwAALI2wAwAALI2wAwAALO3/AwdsNPRjLMSCAAAAAElFTkSuQmCC", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "x = np.arange(3) # change here\n", "width = 0.40\n", "y1 = [\n", " cHouse_read_execution_time,\n", " psql_read_execution_time,\n", " s3_read_execution_time,\n", "] # change here\n", "y2 = [\n", " cHouse_write_execution_time,\n", " psql_write_execution_time,\n", " s3_write_execution_time,\n", "] # change here\n", "plt.bar(x - 0.2, y1, width)\n", "plt.bar(x + 0.2, y2, width)\n", "plt.xticks(x, [\"Click House\", \"Postgresql\", \"S3 Parquet\"])\n", "plt.xlabel(\"Databases\")\n", "plt.ylabel(\"Seconds\")\n", "plt.legend([\"Read\", \"Write\"]) # ver\n", "plt.show()" ] }, { "cell_type": "code", "execution_count": null, "id": "cac1c82a-bcab-47b1-b302-77d1341e5304", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.11" }, "widgets": { "application/vnd.jupyter.widget-state+json": { "state": {}, "version_major": 2, "version_minor": 0 } } }, "nbformat": 4, "nbformat_minor": 5 }