{ "cells": [ { "cell_type": "markdown", "id": "7f1f5a4a-ae34-4a27-9efa-399edc0e384a", "metadata": { "tags": [] }, "source": [ "## Benchmark: ClickHouse Vs. InfluxDB Vs. Postgresql Vs. Parquet \n", "\n", "-----\n", "\n", "#### How to use:\n", "* Rename the file \"properties-model.ini\" to \"properties.ini\"\n", "* Fill with your own credentials\n", "----\n", "\n", "The proposal of this work is to compare the speed in read/writing a midle level of data ( a dataset with 9 columns and 50.000 lines) to four diferent databases:\n", "* ClickHouse\n", "* InfluxDB\n", "* Postgresql\n", "* Parquet (in a S3 Minio Storage)
\n", "ToDo:
\n", "* DuckDB with Polars\n", "* MongoDB\n", "* Kdb+\n", "\n", " \n", "Deve-se relevar:\n", "é uma \"cold-storage\" ou \"frezze-storage\"?
\n", "influxdb: alta leitura e possui a vantagem da indexaçõa para vizualização de dados em gráficos.\n", "\n", "notas: \n", "* comparar tamanho do csv com parquet" ] }, { "cell_type": "markdown", "id": "6bb26ce7-1e84-4665-accd-916bb977f95d", "metadata": { "tags": [] }, "source": [ "### Imports " ] }, { "cell_type": "code", "execution_count": 12, "id": "ab6c6c81-6ac1-4668-a79b-a9a0341fb35a", "metadata": { "tags": [] }, "outputs": [ { "data": { "text/plain": [ "False" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import configparser\n", "\n", "# import pymongo\n", "import io\n", "import time\n", "import timeit\n", "from datetime import datetime\n", "\n", "import duckdb\n", "import influxdb_client\n", "import matplotlib.pyplot as plt\n", "import numpy as np\n", "import pandas as pd\n", "from clickhouse_driver import Client\n", "from dotenv import load_dotenv\n", "from minio import Minio\n", "from pymongo import MongoClient\n", "from pytz import timezone\n", "from sqlalchemy import create_engine\n", "\n", "load_dotenv()" ] }, { "cell_type": "code", "execution_count": 2, "id": "55c3cd57-0996-4723-beb5-8f3196c96009", "metadata": { "tags": [] }, "outputs": [], "source": [ "# Variables\n", "dbname = \"EURUSDtest\"" ] }, { "cell_type": "code", "execution_count": 3, "id": "968403e3-2e5e-4834-b969-be4600e2963a", "metadata": { "tags": [] }, "outputs": [], "source": [ "arq = configparser.RawConfigParser()\n", "arq.read(\"properties.ini\")\n", "ClickHouseUser = arq.get(\"CLICKHOUSE\", \"user\")\n", "ClickHouseKey = arq.get(\"CLICKHOUSE\", \"key\")\n", "ClickHouseUrl = arq.get(\"CLICKHOUSE\", \"url\")\n", "\n", "InfluxDBUser = arq.get(\"INFLUXDB\", \"user\")\n", "InfluxDBKey = arq.get(\"INFLUXDB\", \"key\")\n", "InfluxDBUrl = arq.get(\"INFLUXDB\", \"url\")\n", "InfluxDBBucket = arq.get(\"INFLUXDB\", \"bucket\")\n", "\n", "PostgresqlUser = arq.get(\"POSTGRESQL\", \"user\")\n", "PostgresqlKey = arq.get(\"POSTGRESQL\", \"key\")\n", "PostgresqlUrl = arq.get(\"POSTGRESQL\", \"url\")\n", "PostgresqlDB = arq.get(\"POSTGRESQL\", \"database\")\n", "\n", "S3MinioUser = arq.get(\"S3MINIO\", \"user\")\n", "S3MinioKey = arq.get(\"S3MINIO\", \"key\")\n", "S3MinioUrl = arq.get(\"S3MINIO\", \"url\")\n", "S3MinioRegion = arq.get(\"S3MINIO\", \"region\")\n", "\n", "MongoUser = arq.get(\"MONGODB\", \"user\")\n", "MongoKey = arq.get(\"MONGODB\", \"key\")\n", "MongoUrl = arq.get(\"MONGODB\", \"url\")" ] }, { "cell_type": "code", "execution_count": 4, "id": "3634a4ec-04c2-4f1e-8659-5d22eb17a254", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
idfromattoopencloseminmaxvolume
077308012023-01-02 15:58:4516726751400000000002023-01-02 15:59:001.0659951.0660351.0659301.06607057
177308022023-01-02 15:59:0016726751550000000002023-01-02 15:59:151.0660551.0660851.0660051.06611552
277308032023-01-02 15:59:1516726751700000000002023-01-02 15:59:301.0660801.0660251.0660251.06611057
377308042023-01-02 15:59:3016726751850000000002023-01-02 15:59:451.0659801.0659851.0658851.06604564
477308052023-01-02 15:59:4516726752000000000002023-01-02 16:00:001.0659751.0660551.0658301.06605550
\n", "
" ], "text/plain": [ " id from at to \\\n", "0 7730801 2023-01-02 15:58:45 1672675140000000000 2023-01-02 15:59:00 \n", "1 7730802 2023-01-02 15:59:00 1672675155000000000 2023-01-02 15:59:15 \n", "2 7730803 2023-01-02 15:59:15 1672675170000000000 2023-01-02 15:59:30 \n", "3 7730804 2023-01-02 15:59:30 1672675185000000000 2023-01-02 15:59:45 \n", "4 7730805 2023-01-02 15:59:45 1672675200000000000 2023-01-02 16:00:00 \n", "\n", " open close min max volume \n", "0 1.065995 1.066035 1.065930 1.066070 57 \n", "1 1.066055 1.066085 1.066005 1.066115 52 \n", "2 1.066080 1.066025 1.066025 1.066110 57 \n", "3 1.065980 1.065985 1.065885 1.066045 64 \n", "4 1.065975 1.066055 1.065830 1.066055 50 " ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# %%time\n", "# Load Dataset\n", "df = pd.read_csv(\"out.csv\", index_col=0)\n", "df.head()" ] }, { "cell_type": "code", "execution_count": null, "id": "76199f91-31d6-416b-9f15-5d435b3792c9", "metadata": {}, "outputs": [], "source": [ "df[\"from\"] = pd.to_datetime(df[\"from\"], unit=\"s\")\n", "df[\"to\"] = pd.to_datetime(df[\"to\"], unit=\"s\")\n", "# Optional use when not transoformed yet\n", "# Transform Datetime" ] }, { "cell_type": "markdown", "id": "274cc026-2f48-4e38-b80f-b1a9ff982060", "metadata": { "tags": [] }, "source": [ "#### Funçoes\n", "\n", "-> Class" ] }, { "cell_type": "code", "execution_count": null, "id": "27de1ec8-4de1-440a-b555-b4a46c5ef7ce", "metadata": {}, "outputs": [], "source": [ "def timestamp2dataHora(x, timezone_=\"America/Sao_Paulo\"):\n", " d = datetime.fromtimestamp(x, tz=timezone(timezone_))\n", " return d" ] }, { "cell_type": "markdown", "id": "4a8d5703-9bc9-4d38-83ff-457159304d58", "metadata": { "tags": [] }, "source": [ "### ClickHouse" ] }, { "cell_type": "code", "execution_count": 22, "id": "c3202bbb-2655-45b2-b166-9f45a3ef854c", "metadata": { "tags": [] }, "outputs": [ { "data": { "text/plain": [ "'Database created'" ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# !! driver tcp.\n", "def cHouseConnect():\n", " client = Client(\n", " host=ClickHouseUrl,\n", " user=ClickHouseUser,\n", " password=ClickHouseKey,\n", " settings={\"use_numpy\": True},\n", " )\n", " return client\n", "\n", "\n", "# Create Tables in ClickHouse\n", "# !! ALTERAR TIPOS !!\n", "# ENGINE: 'Memory' desaparece quando server é reiniciado\n", "def cHouseCreateDb(databasename):\n", " client = cHouseConnect()\n", " client.execute(\n", " \"CREATE TABLE IF NOT EXISTS {} (id UInt32,\"\n", " \"from DateTime, at UInt64, to DateTime, open Float64,\"\n", " \"close Float64, min Float64, max Float64, volume UInt32)\"\n", " \"ENGINE MergeTree ORDER BY to\".format(databasename)\n", " )\n", " client.disconnect()\n", " return \"Database created\"\n", "\n", "\n", "# Write dataframe to db\n", "def cHouseInsertDf(dbName, dataframe):\n", " client = cHouseConnect()\n", " client.insert_dataframe(\"INSERT INTO {} VALUES\".format(dbName), dataframe)\n", " client.disconnect()\n", " return \" dataframe {} inserted in clickhouse database\".format(dataframe)\n", "\n", "\n", "def cHouseQueryDf(databaseName):\n", " client = cHouseConnect()\n", " dfQuery = client.query_dataframe(\n", " \"SELECT * FROM default.{}\".format(databaseName)\n", " ) # LIMIT 10000\n", " client.disconnect()\n", " return dfQuery\n", "\n", "\n", "cHouseCreateDb(dbname)" ] }, { "cell_type": "code", "execution_count": 23, "id": "cc4865b3-a1bc-4a35-9624-15334754b3a1", "metadata": {}, "outputs": [], "source": [ "# Insert to db and benchmark time\n", "start = timeit.default_timer()\n", "cHouseInsertDf(dbname, df)\n", "stop = timeit.default_timer()\n", "cHouse_write_execution_time = stop - start" ] }, { "cell_type": "code", "execution_count": 24, "id": "1fac82c1-2d04-44ef-893a-dc13b755e6d8", "metadata": {}, "outputs": [], "source": [ "# read from db and benchmark time\n", "start = timeit.default_timer()\n", "dfCh = cHouseQueryDf(dbname)\n", "stop = timeit.default_timer()\n", "cHouse_read_execution_time = stop - start" ] }, { "cell_type": "code", "execution_count": 25, "id": "597ae7bd-2eea-44d7-b379-f0eb7e745c15", "metadata": { "tags": [] }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
idfromattoopencloseminmaxvolume
077308012023-01-02 15:58:4516726751400000000002023-01-02 15:59:001.0659951.0660351.0659301.06607057
177308012023-01-02 15:58:4516726751400000000002023-01-02 15:59:001.0659951.0660351.0659301.06607057
277308022023-01-02 15:59:0016726751550000000002023-01-02 15:59:151.0660551.0660851.0660051.06611552
377308022023-01-02 15:59:0016726751550000000002023-01-02 15:59:151.0660551.0660851.0660051.06611552
477308032023-01-02 15:59:1516726751700000000002023-01-02 15:59:301.0660801.0660251.0660251.06611057
\n", "
" ], "text/plain": [ " id from at to \\\n", "0 7730801 2023-01-02 15:58:45 1672675140000000000 2023-01-02 15:59:00 \n", "1 7730801 2023-01-02 15:58:45 1672675140000000000 2023-01-02 15:59:00 \n", "2 7730802 2023-01-02 15:59:00 1672675155000000000 2023-01-02 15:59:15 \n", "3 7730802 2023-01-02 15:59:00 1672675155000000000 2023-01-02 15:59:15 \n", "4 7730803 2023-01-02 15:59:15 1672675170000000000 2023-01-02 15:59:30 \n", "\n", " open close min max volume \n", "0 1.065995 1.066035 1.065930 1.066070 57 \n", "1 1.065995 1.066035 1.065930 1.066070 57 \n", "2 1.066055 1.066085 1.066005 1.066115 52 \n", "3 1.066055 1.066085 1.066005 1.066115 52 \n", "4 1.066080 1.066025 1.066025 1.066110 57 " ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dfCh.head()" ] }, { "cell_type": "code", "execution_count": 26, "id": "86794e47-611f-4ca8-a7e8-07e71afafe67", "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "10.64297915700081\n" ] } ], "source": [ "print(cHouse_read_execution_time)" ] }, { "cell_type": "code", "execution_count": 27, "id": "e7926062-8e84-4d3f-90a9-32807ce4f3d4", "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "6.190685558998666\n" ] } ], "source": [ "print(cHouse_write_execution_time)" ] }, { "cell_type": "code", "execution_count": 28, "id": "8faa5683-a204-461d-80c3-67644aa714ce", "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "CPU times: user 2.21 s, sys: 383 ms, total: 2.6 s\n", "Wall time: 10.7 s\n" ] } ], "source": [ "%%time\n", "dfCh = cHouseQueryDf(dbname)" ] }, { "cell_type": "markdown", "id": "1d389546-911f-43f7-aad1-49f7bcc83503", "metadata": { "jp-MarkdownHeadingCollapsed": true, "tags": [] }, "source": [ "### InfluxDB\n" ] }, { "cell_type": "code", "execution_count": null, "id": "c3e7ebfd-76f1-4ac4-9833-312eb1a531af", "metadata": {}, "outputs": [], "source": [ "client = influxdb_client.InfluxDBClient(\n", " url=InfluxDBUrl, token=InfluxDBKey, org=InfluxDBUser\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "cbf61f12-830b-4c57-804a-2257d8b3599a", "metadata": { "tags": [] }, "outputs": [], "source": [ "# Read data from CSV without index and parse 'TimeStamp' as date.\n", "df = pd.read_csv(\"out.csv\", sep=\",\", index_col=False, parse_dates=[\"from\"])\n", "# Set 'TimeStamp' field as index of dataframe # test another indexs\n", "df.set_index(\"from\", inplace=True)" ] }, { "cell_type": "code", "execution_count": null, "id": "54342a28-ba2b-4ade-a692-00566b53a639", "metadata": { "tags": [] }, "outputs": [], "source": [ "df.head()" ] }, { "cell_type": "code", "execution_count": null, "id": "f861fab2-f1b1-49dd-b758-12d10aef3462", "metadata": {}, "outputs": [], "source": [ "%%time\n", "# gravando... demorou... mas deu certo\n", "with client.write_api() as writer:\n", " writer.write(\n", " bucket=InfluxDBBucket,\n", " record=df,\n", " data_frame_measurement_name=\"id\",\n", " data_frame_tag_columns=[\"volume\"],\n", " )" ] }, { "cell_type": "code", "execution_count": null, "id": "0bb2563d-68e2-4ff4-8842-70ac730dc6b1", "metadata": {}, "outputs": [], "source": [ "# data\n", "# |> pivot(\n", "# rowKey:[\"_time\"],\n", "# columnKey: [\"_field\"],\n", "# valueColumn: \"_value\"\n", "# )" ] }, { "cell_type": "code", "execution_count": null, "id": "bb1596f9-4cee-4642-803a-ee61c9dddf64", "metadata": {}, "outputs": [], "source": [ "# Read" ] }, { "cell_type": "markdown", "id": "b9ddfdc6-c899-4f6c-9b4e-8ec6ab6d7e05", "metadata": { "tags": [] }, "source": [ "### Postgresql" ] }, { "cell_type": "code", "execution_count": 7, "id": "16cd8eb7-333d-43fd-88e0-ee983645d3fd", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Engine(postgresql+psycopg2://postgres:***@192.168.1.133:5432/postgres)" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Connect / Create Tables\n", "def psqlConnect():\n", " engine = create_engine(\n", " \"postgresql+psycopg2://{}:{}@{}:5432/{}\".format(\n", " PostgresqlUser, PostgresqlKey, PostgresqlUrl, PostgresqlDB\n", " )\n", " )\n", " return engine\n", "\n", "\n", "psqlConnect()\n", "# testar função" ] }, { "cell_type": "code", "execution_count": null, "id": "e173a45b-60a1-4c33-946e-ccf98bf8e97f", "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": 18, "id": "be31f3a0-b7ed-48e6-9b65-dc16319fb8d1", "metadata": {}, "outputs": [], "source": [ "# Drop old table and create new empty table\n", "def psqlCreateTables(databaseName):\n", " engine = psqlConnect()\n", " df.head(0).to_sql(databaseName, engine, if_exists=\"replace\", index=False)\n", " # Write\n", " conn = engine.raw_connection()\n", " cur = conn.cursor()\n", " output = io.StringIO()\n", " df.to_csv(output, sep=\"\\t\", header=False, index=False)\n", " output.seek(0)\n", " contents = output.getvalue()\n", "\n", " cur.copy_from(output, \"comparedbs\") # , null=\"\") # null values become ''\n", " conn.commit()\n", " cur.close()\n", " conn.close()\n", " # disconnect()\n", " return 0\n", "\n", "\n", "# funcao read sql\n", "def psqlReadTables():\n", " engine = psqlConnect()\n", " df = pd.read_sql_query('select * from \"comparedbs\"', con=engine)\n", " return df\n", "\n", "\n", "# testar função" ] }, { "cell_type": "code", "execution_count": 19, "id": "98cc9360-4b84-43e4-b23b-b32d0c50c3b9", "metadata": { "tags": [] }, "outputs": [], "source": [ "# Insert to db and benchmark time\n", "start = timeit.default_timer()\n", "psqlCreateTables(dbname)\n", "stop = timeit.default_timer()\n", "psql_write_execution_time = stop - start" ] }, { "cell_type": "code", "execution_count": 30, "id": "82e1b44e-35af-403f-9936-5e1561fd5abf", "metadata": { "tags": [] }, "outputs": [], "source": [ "start = timeit.default_timer()\n", "psqlReadTables()\n", "stop = timeit.default_timer()\n", "psql_read_execution_time = stop - start" ] }, { "cell_type": "code", "execution_count": 10, "id": "a7883c4d-4609-4380-8a45-246b7ca2f9c5", "metadata": { "tags": [] }, "outputs": [ { "ename": "NameError", "evalue": "name 'engine' is not defined", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mNameError\u001b[0m Traceback (most recent call last)", "File \u001b[0;32m:2\u001b[0m\n", "\u001b[0;31mNameError\u001b[0m: name 'engine' is not defined" ] } ], "source": [ "# %%time\n", "# # Write\n", "# conn = engine.raw_connection()\n", "# cur = conn.cursor()\n", "# output = io.StringIO()\n", "# df.to_csv(output, sep=\"\\t\", header=False, index=False)\n", "# output.seek(0)\n", "# contents = output.getvalue()\n", "\n", "# cur.copy_from(output, \"comparedbs\") # , null=\"\") # null values become ''\n", "# conn.commit()\n", "# cur.close()\n", "# conn.close()" ] }, { "cell_type": "code", "execution_count": null, "id": "73de4294-1284-49b0-b31e-45db6e835877", "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": null, "id": "e37a93e1-fc0e-4d27-9e16-dca6c8aea324", "metadata": {}, "outputs": [], "source": [ "start = time.time()\n", "# %%time\n", "# Read\n", "df = pd.read_sql_query('select * from \"comparedbs\"', con=engine)\n", "end = time.time()\n", "postgresql_read_time = exec_time(start, end)" ] }, { "cell_type": "code", "execution_count": null, "id": "6d1b7480-5bc7-4f08-8cf3-b9590802d8f7", "metadata": { "tags": [] }, "outputs": [], "source": [ "print(postgresql_read_time)" ] }, { "cell_type": "code", "execution_count": null, "id": "6acb2959-3255-43bd-aea5-9ef70acc8902", "metadata": { "tags": [] }, "outputs": [], "source": [ "df.head()" ] }, { "cell_type": "markdown", "id": "f9e0393d-7d1d-406a-a068-9dbf4968e977", "metadata": { "tags": [] }, "source": [ "### S3 Parquet" ] }, { "cell_type": "code", "execution_count": null, "id": "60a990e2-4607-4654-84ec-17d4985adae2", "metadata": { "tags": [] }, "outputs": [], "source": [ "# fazer sem funçao para ver se melhora\n", "# verifique se esta no ssd os arquivos da pasta git\n", "def main():\n", " client = Minio(\n", " S3MinioUrl,\n", " secure=False,\n", " region=S3MinioRegion,\n", " access_key=\"MatMPA7NyHltz7DQ\",\n", " secret_key=\"SO1IG5iBPSjNPZanYUaHCLcoSbjphLCP\",\n", " )\n", "\n", " # Make bucket if not exist.\n", " found = client.bucket_exists(\"data\")\n", " if not found:\n", " client.make_bucket(\"data\")\n", " else:\n", " print(\"Bucket 'data' already exists\")\n", "\n", " # Upload\n", " client.fput_object(\n", " \"data\",\n", " \"data.parquet\",\n", " \"data/data.parquet\",\n", " )\n", " # print(\n", " # \"'data/data.parquet' is successfully uploaded as \"\n", " # \"object 'data.parquet' to bucket 'data'.\"\n", " # )" ] }, { "cell_type": "code", "execution_count": null, "id": "390918c8-c88f-404a-96c4-685d578fdad0", "metadata": { "tags": [] }, "outputs": [], "source": [ "%%time\n", "df.to_parquet(\"data/data.parquet\")\n", "if __name__ == \"__main__\":\n", " try:\n", " main()\n", " except S3Error as exc:\n", " print(\"error occurred.\", exc)" ] }, { "cell_type": "code", "execution_count": null, "id": "a9e07143-8c11-4b68-a869-c3922cda9092", "metadata": { "tags": [] }, "outputs": [], "source": [ "pq = pd.read_parquet(\"data/data.parquet\", engine=\"pyarrow\")\n", "pq.head()" ] }, { "cell_type": "markdown", "id": "50d1fc58-89a7-4507-aff0-6e943656cfe0", "metadata": { "tags": [] }, "source": [ "### MongoDB" ] }, { "cell_type": "code", "execution_count": null, "id": "d104d9af-fa34-4261-8478-329a28ee4f2e", "metadata": { "tags": [] }, "outputs": [], "source": [ "# Load csv dataset\n", "data = pd.read_csv(\"out.csv\")" ] }, { "cell_type": "code", "execution_count": null, "id": "0af8f72c-5b58-4dfc-af36-c5b4bc79f127", "metadata": { "tags": [] }, "outputs": [], "source": [ "# Connect to MongoDB\n", "client = MongoClient(\n", " # \"mongodb://192.168.1.133:27017\"\n", " \"mongodb://{}:{}@{}/EURUSDtest?retryWrites=true&w=majority\".format(\n", " MongoUser, MongoKey, MongoUrl\n", " ),\n", " authSource=\"admin\",\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "f1b20d15-f5af-463c-813f-ffae61119de1", "metadata": { "tags": [] }, "outputs": [], "source": [ "db = client[\"EUROUSDtest\"]\n", "collection = db[\"finance\"]\n", "# data.reset_index(inplace=True)\n", "data_dict = data.to_dict(\"records\")" ] }, { "cell_type": "code", "execution_count": null, "id": "70674d23-f375-4659-87ec-c745dec96d54", "metadata": { "tags": [] }, "outputs": [], "source": [ "%%time\n", "# Insert collection\n", "collection.insert_many(data_dict)" ] }, { "cell_type": "code", "execution_count": null, "id": "81a4a33d-5914-45d8-af4e-2b0aabd2ac38", "metadata": { "tags": [] }, "outputs": [], "source": [ "# read" ] }, { "cell_type": "markdown", "id": "97405e42-61dc-42c7-8220-237a312c0ec7", "metadata": { "tags": [] }, "source": [ "### DuckDB" ] }, { "cell_type": "code", "execution_count": null, "id": "bbcdb883-d6dc-46db-88db-4c90b84522ba", "metadata": {}, "outputs": [], "source": [ "cursor = duckdb.connect()\n", "print(cursor.execute(\"SELECT 42\").fetchall())" ] }, { "cell_type": "code", "execution_count": null, "id": "35025a6e-9dc7-46cf-a792-76b3d84f1ac0", "metadata": { "tags": [] }, "outputs": [], "source": [ "%%time\n", "conn = duckdb.connect()\n", "data = pd.read_csv(\"out.csv\")\n", "conn.register(\"EURUSDtest\", data)" ] }, { "cell_type": "code", "execution_count": null, "id": "c6abdaaa-3ac2-425b-9208-d6cb79afe966", "metadata": { "tags": [] }, "outputs": [], "source": [ "display(conn.execute(\"SHOW TABLES\").df())" ] }, { "cell_type": "code", "execution_count": null, "id": "2acce0f3-f0b2-47d0-8e0d-f9e9687efc18", "metadata": { "tags": [] }, "outputs": [], "source": [ "%%time\n", "df = conn.execute(\"SELECT * FROM EURUSDtest\").df()\n", "df" ] }, { "cell_type": "markdown", "id": "4409cc89-ed14-4313-ac89-65b826038533", "metadata": { "tags": [] }, "source": [ "### Kdb+" ] }, { "cell_type": "code", "execution_count": null, "id": "14f63810-1943-4e28-9bce-2148be6be02d", "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "\n", "np.bool = np.bool_\n", "from qpython import qconnection" ] }, { "cell_type": "code", "execution_count": null, "id": "8ff6c090-7e02-435a-a179-f2aab81da972", "metadata": {}, "outputs": [], "source": [ "# read csv\n", "data = pd.read_csv(\"out.csv\")" ] }, { "cell_type": "code", "execution_count": null, "id": "b4eb8ab9-81e8-4732-8cf7-51f0981d3d57", "metadata": { "tags": [] }, "outputs": [], "source": [ "# open connection\n", "q = qconnection.QConnection(host=\"localhost\", port=5001)\n", "q.open()" ] }, { "cell_type": "code", "execution_count": null, "id": "97cb6b5b-65a5-46a0-a4ee-e5c535a716ab", "metadata": {}, "outputs": [], "source": [ "%%time\n", "# send df to kd+ in memory bank\n", "q.sendSync(\"{t::x}\", data)" ] }, { "cell_type": "code", "execution_count": null, "id": "c2ed2d51-bc8e-4207-892a-35fc55d43570", "metadata": {}, "outputs": [], "source": [ "# write to on disk table\n", "q.sendSync(\"`:/home/sandman/q/tab1 set t\")" ] }, { "cell_type": "code", "execution_count": null, "id": "9c055a95-f73f-43a3-8fbd-61e42235117e", "metadata": { "tags": [] }, "outputs": [], "source": [ "%%time\n", "# read from on disk table\n", "df2 = q.sendSync(\"tab2: get `:/home/sandman/q/tab1\")" ] }, { "cell_type": "code", "execution_count": null, "id": "9760de38-9f04-4322-bfff-c7ee12d5dee5", "metadata": { "tags": [] }, "outputs": [], "source": [ "# print(df2)" ] }, { "cell_type": "code", "execution_count": null, "id": "c06c9222-c69d-4872-9d21-052281a013e2", "metadata": { "tags": [] }, "outputs": [], "source": [ "%%time\n", "# load to variable df2\n", "df2 = q.sendSync(\"tab2\")" ] }, { "cell_type": "code", "execution_count": null, "id": "8815f01c-fd0a-4f94-ab7f-f8ede84ba4e7", "metadata": { "tags": [] }, "outputs": [], "source": [ "# df2(type)" ] }, { "cell_type": "code", "execution_count": null, "id": "e6ed3927-4395-45cd-9a28-88c5db01f2e5", "metadata": { "tags": [] }, "outputs": [], "source": [ "%%time\n", "# converto to dataframe\n", "df = pd.DataFrame(q(\"t\")) # , pandas=True))\n", "df.head()" ] }, { "cell_type": "code", "execution_count": null, "id": "0fc7f16b-6c39-4ebe-88d2-ff857e30ab62", "metadata": { "tags": [] }, "outputs": [], "source": [ "%%time\n", "# select\n", "df3 = q.sendSync(\"select from t\")" ] }, { "cell_type": "code", "execution_count": null, "id": "c88646ca-3d25-4a85-80b5-f9e559f568dd", "metadata": { "tags": [] }, "outputs": [], "source": [ "q.close()" ] }, { "cell_type": "markdown", "id": "7baf1fd1-2afd-41b5-a579-33f053e4ddfc", "metadata": {}, "source": [ "## Graph\n" ] }, { "cell_type": "code", "execution_count": 37, "id": "a9740731-6077-4bf1-bb65-b4c4225ac79b", "metadata": { "tags": [] }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjsAAAGwCAYAAABPSaTdAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA1i0lEQVR4nO3deXQUZb7G8adD9p1EyIIBQkA2kVUwssRAMCAwQXAdFFAGVEDEKJtXGAExgLIIg+I4w+aAKAqoqCyTsC8xJICCCIhsR0mYAZKQAElI6v7Bta4REEg6dFN8P+f0OfRbVW//qrHsh7feqrIZhmEIAADAolwcXQAAAEBFIuwAAABLI+wAAABLI+wAAABLI+wAAABLI+wAAABLI+wAAABLc3V0Ac6gpKREv/zyi/z8/GSz2RxdDgAAuAaGYejMmTMKDw+Xi8uVx28IO5J++eUXRUREOLoMAABQBseOHdPtt99+xeWEHUl+fn6SLn5Z/v7+Dq4GAABci9zcXEVERJi/41dC2JHMU1f+/v6EHQAAbjJXm4LCBGUAAGBphB0AAGBphB0AAGBpzNkBAMCOiouLVVRU5OgyLMHNzU2VKlUqdz+EHQAA7MAwDGVmZio7O9vRpVhKYGCgQkNDy3UfPMIOAAB28GvQqVq1qry9vblJbTkZhqGzZ8/qxIkTkqSwsLAy90XYAQCgnIqLi82gExwc7OhyLMPLy0uSdOLECVWtWrXMp7SYoAwAQDn9OkfH29vbwZVYz6/faXnmQRF2AACwE05d2Z89vlPCDgAAsDTCDgAAsDQmKAMAUIFqjvzyhn3W4YldbthnlVffvn2VnZ2t5cuXV/hnMbIDAMAtrG/fvrLZbLLZbHJzc1NkZKSGDx+u8+fPO7o0u2FkBwCAW1ynTp00d+5cFRUVKT09XX369JHNZtOkSZMcXZpdMLIDAMAtzsPDQ6GhoYqIiFD37t0VFxenNWvWSJJKSkqUlJSkyMhIeXl5qXHjxvrkk0/MbYuLi9WvXz9zed26dfX222+X6r+4uFiJiYkKDAxUcHCwhg8fLsMwbtj+MbIDAHZwI+dl4Opuprkrzmb37t3asmWLatSoIUlKSkrSv/71L82ePVt16tTRhg0b9MQTT6hKlSqKiYlRSUmJbr/9di1ZskTBwcHasmWLBgwYoLCwMD3yyCOSpClTpmjevHmaM2eO6tevrylTpmjZsmVq3779Ddknwg4AALe4FStWyNfXVxcuXFBBQYFcXFz0t7/9TQUFBXrjjTf073//W9HR0ZKkWrVqadOmTXrvvfcUExMjNzc3jR071uwrMjJSW7du1ccff2yGnenTp2vUqFHq0aOHJGn27NlatWrVDds/wg4AALe42NhYvfvuu8rPz9e0adPk6uqqnj17as+ePTp79qw6duxYav3CwkI1bdrUfD9r1izNmTNHR48e1blz51RYWKgmTZpIknJycnT8+HG1atXKXN/V1VUtWrS4YaeyCDsAANzifHx8VLt2bUnSnDlz1LhxY/3zn//UnXfeKUn68ssvVa1atVLbeHh4SJIWL16sl19+WVOmTFF0dLT8/Pz05ptvKjU19cbuxB8g7AAAAJOLi4teeeUVJSYmav/+/fLw8NDRo0cVExNz2fU3b96se++9VwMHDjTbDh48aP45ICBAYWFhSk1NVbt27SRJFy5cUHp6upo1a1axO/N/CDsAAKCUhx9+WMOGDdN7772nl19+WS+++KJKSkrUpk0b5eTkaPPmzfL391efPn1Up04dLViwQKtWrVJkZKQ++OADpaWlKTIy0uzvhRde0MSJE1WnTh3Vq1dPU6dOVXZ29g3bH8IOAAAV6Ga8MszV1VWDBw/W5MmTdejQIVWpUkVJSUn66aefFBgYqGbNmumVV16RJD3zzDPasWOHHn30UdlsNj3++OMaOHCgvv76a7O/l156ScePH1efPn3k4uKip59+Wg8++KBycnJuyP7YjBt5obuTys3NVUBAgHJycuTv7+/ocgDchLj03Lnc6IBx/vx5HTp0SJGRkfL09Lyhn211f/TdXuvvNzcVBAAAlkbYAQAAlkbYAQAAlkbYAQAAlkbYAQAAlkbYAQAAlkbYAQAAlubQsLNhwwZ169ZN4eHhstlsWr58eanlhmFozJgxCgsLk5eXl+Li4nTgwIFS65w6dUq9evWSv7+/AgMD1a9fP+Xl5d3AvQAAAM7MoWEnPz9fjRs31qxZsy67fPLkyZoxY4Zmz56t1NRU+fj4KD4+XufPnzfX6dWrl/bs2aM1a9ZoxYoV2rBhgwYMGHCjdgEAAEiqWbOmpk+f7ugyLsuhj4vo3LmzOnfufNllhmFo+vTpevXVV5WQkCBJWrBggUJCQrR8+XI99thj2rt3r1auXKm0tDS1aNFCkjRz5kw98MADeuuttxQeHn7ZvgsKClRQUGC+z83NtfOeAQDwf14LuIGfdX2PX5g9e7aGDRum06dPy9X1YiTIy8tT5cqV1bp1a61bt85cd926dYqNjdWPP/6oqKioS/pKS0uTj4+P+d5ms2nZsmXq3r17mXbFnpx2zs6hQ4eUmZmpuLg4sy0gIECtWrXS1q1bJUlbt25VYGCgGXQkKS4uTi4uLn/4aPmkpCQFBASYr4iIiIrbEQAAnFRsbKzy8vK0fft2s23jxo0KDQ1VampqqTMpa9euVfXq1S8JOoWFhZKkKlWqyNvb+8YUfp2cNuxkZmZKkkJCQkq1h4SEmMsyMzNVtWrVUstdXV0VFBRkrnM5o0aNUk5Ojvk6duyYnasHAMD51a1bV2FhYZeM4CQkJCgyMlLbtm0r1R4bG6u+ffuqe/fumjBhgsLDw1W3bl1JpU9j1axZU5L04IMPymazme8l6bPPPlOzZs3k6empWrVqaezYsbpw4UKF7qfThp2K5OHhIX9//1IvAABuRbGxsVq7dq35fu3atbrvvvsUExNjtp87d06pqamKjY2VJCUnJ2vfvn3mfNnfS0tLkyTNnTtXx48fN99v3LhRvXv31gsvvKDvv/9e7733nubNm6cJEyZU6D46bdgJDQ2VJGVlZZVqz8rKMpeFhobqxIkTpZZfuHBBp06dMtcBAABXFhsbq82bN+vChQs6c+aMduzYoZiYGLVr184c8dm6dasKCgrMsOPj46N//OMfatiwoRo2bHhJn1WqVJEkBQYGKjQ01Hw/duxYjRw5Un369FGtWrXUsWNHjR8/Xu+9916F7qPThp3IyEiFhoYqOTnZbMvNzVVqaqqio6MlSdHR0crOzlZ6erq5TkpKikpKStSqVasbXjMAADeb++67T/n5+UpLS9PGjRt1xx13qEqVKoqJiTHn7axbt061atVS9erVJUmNGjWSu7v7dX/Wrl27NG7cOPn6+pqv/v376/jx4zp79qy9d83k0Kux8vLy9OOPP5rvDx06pJ07dyooKEjVq1fX0KFD9frrr6tOnTqKjIzU6NGjFR4ebs7srl+/vjp16qT+/ftr9uzZKioq0uDBg/XYY49d8UosAADw/2rXrq3bb79da9eu1enTpxUTEyNJCg8PV0REhLZs2aK1a9eqffv25ja/verqeuTl5Wns2LHq0aPHJcs8PT3LtgPXwKFhZ/v27eaQmCQlJiZKkvr06aN58+Zp+PDhys/P14ABA5Sdna02bdpo5cqVpb6QhQsXavDgwerQoYNcXFzUs2dPzZgx44bvCwAAN6vY2FitW7dOp0+f1rBhw8z2du3a6euvv9Y333yj55577rr6dHNzU3Fxcam2Zs2aad++fapdu7Zd6r5WDg079913nwzDuOJym82mcePGady4cVdcJygoSIsWLaqI8gAAuCXExsZq0KBBKioqMkd2JCkmJkaDBw9WYWFhqcGJa1GzZk0lJyerdevW8vDwUOXKlTVmzBh17dpV1atX10MPPSQXFxft2rVLu3fv1uuvv27v3TI57ZwdAABwY8TGxurcuXOqXbt2qVu+xMTE6MyZM+Yl6tdjypQpWrNmjSIiItS0aVNJUnx8vFasWKHVq1fr7rvv1j333KNp06apRo0adt2f37MZfzS0covIzc1VQECAcnJyuAwdQJnUHPmlo0vAbxye2OWGft758+d16NAhRUZGVujck1vRH3231/r7zcgOAACwNMIOAACwNMIOAACwNMIOAACwNMIOAAB2wjU/9meP75SwAwBAObm5uUlShT7y4Fb163f663dcFg69qSAAAFZQqVIlBQYGmg+n9vb2ls1mc3BVNzfDMHT27FmdOHFCgYGBqlSpUpn7IuwAAGAHoaGhkmQGHtjHr09OLw/CDgAAdmCz2RQWFqaqVauqqKjI0eVYgpubW7lGdH5F2AEAwI4qVapklx9o2A8TlAEAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKU5ddgpLi7W6NGjFRkZKS8vL0VFRWn8+PEyDMNcxzAMjRkzRmFhYfLy8lJcXJwOHDjgwKoBAIAzceqwM2nSJL377rv629/+pr1792rSpEmaPHmyZs6caa4zefJkzZgxQ7Nnz1Zqaqp8fHwUHx+v8+fPO7ByAADgLFwdXcAf2bJlixISEtSlSxdJUs2aNfXhhx/qm2++kXRxVGf69Ol69dVXlZCQIElasGCBQkJCtHz5cj322GOX7begoEAFBQXm+9zc3AreEwAA4ChOPbJz7733Kjk5Wfv375ck7dq1S5s2bVLnzp0lSYcOHVJmZqbi4uLMbQICAtSqVStt3br1iv0mJSUpICDAfEVERFTsjgAAAIdx6pGdkSNHKjc3V/Xq1VOlSpVUXFysCRMmqFevXpKkzMxMSVJISEip7UJCQsxllzNq1CglJiaa73Nzcwk8AABYlFOHnY8//lgLFy7UokWL1LBhQ+3cuVNDhw5VeHi4+vTpU+Z+PTw85OHhYcdKAQCAs3LqsDNs2DCNHDnSnHvTqFEjHTlyRElJSerTp49CQ0MlSVlZWQoLCzO3y8rKUpMmTRxRMgAAcDJOPWfn7NmzcnEpXWKlSpVUUlIiSYqMjFRoaKiSk5PN5bm5uUpNTVV0dPQNrRUAADgnpx7Z6datmyZMmKDq1aurYcOG2rFjh6ZOnaqnn35akmSz2TR06FC9/vrrqlOnjiIjIzV69GiFh4ere/fuji0eAAA4BacOOzNnztTo0aM1cOBAnThxQuHh4XrmmWc0ZswYc53hw4crPz9fAwYMUHZ2ttq0aaOVK1fK09PTgZUDAABnYTN+ezviW1Rubq4CAgKUk5Mjf39/R5cD4CZUc+SXji4Bv3F4YhdHl4Ab4Fp/v516zg4AAEB5EXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAICllSnsnDt3TmfPnjXfHzlyRNOnT9fq1avtVhgAAIA9lCnsJCQkaMGCBZKk7OxstWrVSlOmTFFCQoLeffdduxYIAABQHmUKOxkZGWrbtq0k6ZNPPlFISIiOHDmiBQsWaMaMGXYtEAAAoDzKFHbOnj0rPz8/SdLq1avVo0cPubi46J577tGRI0fsWiAAAEB5lCns1K5dW8uXL9exY8e0atUq3X///ZKkEydOyN/f364FAgAAlEeZws6YMWP08ssvq2bNmmrVqpWio6MlXRzladq0qV0LBAAAKA/Xsmz00EMPqU2bNjp+/LgaN25stnfo0EEPPvig3YoDAAAorzLfZyc0NFRNmzaVi8v/d9GyZUvVq1fPLoX96ueff9YTTzyh4OBgeXl5qVGjRtq+fbu53DAMjRkzRmFhYfLy8lJcXJwOHDhg1xoAAMDN65pHdnr06HHNnS5durRMxfze6dOn1bp1a8XGxurrr79WlSpVdODAAVWuXNlcZ/LkyZoxY4bmz5+vyMhIjR49WvHx8fr+++/l6elplzoAAMDN65rDTkBAgPlnwzC0bNkyBQQEqEWLFpKk9PR0ZWdnX1couppJkyYpIiJCc+fONdsiIyNL1TF9+nS9+uqrSkhIkCQtWLBAISEhWr58uR577DG71QIAAG5O1xx2fhs4RowYoUceeUSzZ89WpUqVJEnFxcUaOHCgXa/G+vzzzxUfH6+HH35Y69evV7Vq1TRw4ED1799fknTo0CFlZmYqLi7O3CYgIECtWrXS1q1brxh2CgoKVFBQYL7Pzc21W80AAMC5lGnOzpw5c/Tyyy+bQUeSKlWqpMTERM2ZM8duxf3000969913VadOHa1atUrPPfechgwZovnz50uSMjMzJUkhISGltgsJCTGXXU5SUpICAgLMV0REhN1qBgAAzqVMYefChQv64YcfLmn/4YcfVFJSUu6iflVSUqJmzZrpjTfeUNOmTTVgwAD1799fs2fPLle/o0aNUk5Ojvk6duyYnSoGAADOpkyXnj/11FPq16+fDh48qJYtW0qSUlNTNXHiRD311FN2Ky4sLEwNGjQo1Va/fn19+umnki5eESZJWVlZCgsLM9fJyspSkyZNrtivh4eHPDw87FYnAABwXmUKO2+99ZZCQ0M1ZcoUHT9+XNLFYDJs2DC99NJLdiuudevW2rdvX6m2/fv3q0aNGpIuTlYODQ1VcnKyGW5yc3OVmpqq5557zm51AACAm1eZwo6Li4uGDx+u4cOHm5N7K+IxES+++KLuvfdevfHGG3rkkUf0zTff6O9//7v+/ve/S5JsNpuGDh2q119/XXXq1DEvPQ8PD1f37t3tXg8AALj5lCns/FZFPgvr7rvv1rJlyzRq1CiNGzdOkZGRmj59unr16mWuM3z4cOXn52vAgAHKzs5WmzZttHLlSu6xAwAAJEk2wzCM690oKytLL7/8spKTk3XixAn9vovi4mK7FXgj5ObmKiAgQDk5OTzIFECZ1Bz5paNLwG8cntjF0SXgBrjW3+8yjez07dtXR48e1ejRoxUWFiabzVbmQgEAACpSmcLOpk2btHHjxj+84gkAAMAZlOk+OxEREZecugIAAHBGZQo706dP18iRI3X48GE7lwMAAGBfZTqN9eijj+rs2bOKioqSt7e33NzcSi0/deqUXYoDAAAorzKFnenTp9u5DAAAgIpRprDTp08fe9cBAABQIcp8U8Hi4mItX75ce/fulSQ1bNhQf/rTn0o9CR0AAMDRyhR2fvzxRz3wwAP6+eefVbduXUlSUlKSIiIi9OWXXyoqKsquRQIAAJRVma7GGjJkiKKionTs2DFlZGQoIyNDR48eVWRkpIYMGWLvGgEAAMqsTCM769ev17Zt2xQUFGS2BQcHa+LEiWrdurXdigMAACivMo3seHh46MyZM5e05+Xlyd3dvdxFAQAA2EuZwk7Xrl01YMAApaamyjAMGYahbdu26dlnn9Wf/vQne9cIAABQZmUKOzNmzFBUVJSio6Pl6ekpT09PtW7dWrVr19bbb79t7xoBAADKrExzdgIDA/XZZ5/pxx9/NC89r1+/vmrXrm3X4gAAAMqrzPfZkaTatWsTcAAAgFMr02msnj17atKkSZe0T548WQ8//HC5iwIAALCXMoWdDRs26IEHHrikvXPnztqwYUO5iwIAALCXMoWdK11i7ubmptzc3HIXBQAAYC9lCjuNGjXSRx99dEn74sWL1aBBg3IXBQAAYC9lmqA8evRo9ejRQwcPHlT79u0lScnJyfrwww+1ZMkSuxYIAABQHmUKO926ddPy5cv1xhtv6JNPPpGXl5fuuusu/fvf/1ZMTIy9awQAACizMl963qVLF3Xp0sWetQAAANhdmebsSFJ2drb+8Y9/6JVXXtGpU6ckSRkZGfr555/tVhwAAEB5lWlk59tvv1VcXJwCAgJ0+PBh/eUvf1FQUJCWLl2qo0ePasGCBfauEwAAoEzKNLKTmJiovn376sCBA/L09DTbH3jgAe6zAwAAnEqZwk5aWpqeeeaZS9qrVaumzMzMchcFAABgL2UKOx4eHpe9eeD+/ftVpUqVchcFAABgL2UKO3/60580btw4FRUVSZJsNpuOHj2qESNGqGfPnnYtEAAAoDzKFHamTJmivLw8Va1aVefOnVNMTIyioqLk6+urCRMm2LtGAACAMivT1VgBAQFas2aNNm3apG+//VZ5eXlq3ry5OnToYO/6AAAAyuW6Rna2bt2qFStWmO/btGkjHx8fvfPOO3r88cc1YMAAFRQU2L1IAACAsrqusDNu3Djt2bPHfP/dd9+pf//+6tixo0aOHKkvvvhCSUlJdi8SAACgrK4r7OzcubPUqarFixerZcuWev/995WYmKgZM2bo448/tnuRAAAAZXVdYef06dMKCQkx369fv16dO3c239999906duyY/aoDAAAop+sKOyEhITp06JAkqbCwUBkZGbrnnnvM5WfOnJGbm5t9KwQAACiH6wo7DzzwgEaOHKmNGzdq1KhR8vb2Vtu2bc3l3377raKiouxeJAAAQFld16Xn48ePV48ePRQTEyNfX1/Nnz9f7u7u5vI5c+bo/vvvt3uRAAAAZXVdYee2227Thg0blJOTI19fX1WqVKnU8iVLlsjX19euBQIAAJRHmW8qeDlBQUHlKgYAAMDeyvS4CAAAgJsFYQcAAFgaYQcAAFgaYQcAAFgaYQcAAFgaYQcAAFgaYQcAAFgaYQcAAFgaYQcAAFgaYQcAAFgaYQcAAFgaYQcAAFgaYQcAAFgaYQcAAFgaYQcAAFgaYQcAAFjaTRV2Jk6cKJvNpqFDh5pt58+f16BBgxQcHCxfX1/17NlTWVlZjisSAAA4lZsm7KSlpem9997TXXfdVar9xRdf1BdffKElS5Zo/fr1+uWXX9SjRw8HVQkAAJzNTRF28vLy1KtXL73//vuqXLmy2Z6Tk6N//vOfmjp1qtq3b6/mzZtr7ty52rJli7Zt23bF/goKCpSbm1vqBQAArOmmCDuDBg1Sly5dFBcXV6o9PT1dRUVFpdrr1aun6tWra+vWrVfsLykpSQEBAeYrIiKiwmoHAACO5fRhZ/HixcrIyFBSUtIlyzIzM+Xu7q7AwMBS7SEhIcrMzLxin6NGjVJOTo75OnbsmL3LBgAATsLV0QX8kWPHjumFF17QmjVr5Onpabd+PTw85OHhYbf+AACA83LqkZ309HSdOHFCzZo1k6urq1xdXbV+/XrNmDFDrq6uCgkJUWFhobKzs0ttl5WVpdDQUMcUDQAAnIpTj+x06NBB3333Xam2p556SvXq1dOIESMUEREhNzc3JScnq2fPnpKkffv26ejRo4qOjnZEyQAAwMk4ddjx8/PTnXfeWarNx8dHwcHBZnu/fv2UmJiooKAg+fv76/nnn1d0dLTuueceR5QMAACcjFOHnWsxbdo0ubi4qGfPniooKFB8fLzeeecdR5cFAACchM0wDMPRRThabm6uAgIClJOTI39/f0eXA+AmVHPkl44uAb9xeGIXR5eAG+Baf7+deoIyAABAeRF2AACApRF2AACApRF2AACApRF2AACApRF2AACApRF2AACApRF2AACApRF2AACApRF2AACApRF2AACApRF2AACApRF2AACApRF2AACApRF2AACApRF2AACApRF2AACApRF2AACApRF2AACApRF2AACApRF2AACApRF2AACApRF2AACApRF2AACApRF2AACApRF2AACApRF2AACApRF2AACApRF2AACApRF2AACApRF2AACApRF2AACApRF2AACApRF2AACApRF2AACApRF2AACApRF2AACApRF2AACApRF2AACApRF2AACApRF2AACApRF2AACApRF2AACApRF2AACApRF2AACApRF2AACApRF2AACApRF2AACApRF2AACApRF2AACApRF2AACApRF2AACApRF2AACApRF2AACApbk6ugAAAOzutQBHV4Dfei3HoR/v1CM7SUlJuvvuu+Xn56eqVauqe/fu2rdvX6l1zp8/r0GDBik4OFi+vr7q2bOnsrKyHFQxAABwNk4ddtavX69BgwZp27ZtWrNmjYqKinT//fcrPz/fXOfFF1/UF198oSVLlmj9+vX65Zdf1KNHDwdWDQAAnIlTn8ZauXJlqffz5s1T1apVlZ6ernbt2iknJ0f//Oc/tWjRIrVv316SNHfuXNWvX1/btm3TPffc44iyAQCAE3HqkZ3fy8m5eM4vKChIkpSenq6ioiLFxcWZ69SrV0/Vq1fX1q1br9hPQUGBcnNzS70AAIA13TRhp6SkREOHDlXr1q115513SpIyMzPl7u6uwMDAUuuGhIQoMzPzin0lJSUpICDAfEVERFRk6QAAwIFumrAzaNAg7d69W4sXLy53X6NGjVJOTo75OnbsmB0qBAAAzsip5+z8avDgwVqxYoU2bNig22+/3WwPDQ1VYWGhsrOzS43uZGVlKTQ09Ir9eXh4yMPDoyJLBgAATsKpR3YMw9DgwYO1bNkypaSkKDIystTy5s2by83NTcnJyWbbvn37dPToUUVHR9/ocgEAgBNy6pGdQYMGadGiRfrss8/k5+dnzsMJCAiQl5eXAgIC1K9fPyUmJiooKEj+/v56/vnnFR0dzZVYAABAkpOHnXfffVeSdN9995Vqnzt3rvr27StJmjZtmlxcXNSzZ08VFBQoPj5e77zzzg2uFAAAOCunDjuGYVx1HU9PT82aNUuzZs26ARUBAICbjVPP2QEAACgvwg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0V0cXYHU1R37p6BLwG4cndnF0CQCAG4yRHQAAYGmEHQAAYGmEHQAAYGmEHQAAYGmEHQAAYGmEHQAAYGmEHQAAYGmEHQAAYGmEHQAAYGmEHQAAYGmEHQAAYGmEHQAAYGmEHQAAYGmEHQAAYGmuji4AuKFeC3B0Bfi913IcXQEAi2NkBwAAWBphBwAAWBphBwAAWBphBwAAWBphBwAAWBphBwAAWBphBwAAWBphBwAAWBphBwAAWBphBwAAWBphBwAAWBphBwAAWBphBwAAWBphBwAAWBphBwAAWBphBwAAWJplws6sWbNUs2ZNeXp6qlWrVvrmm28cXRIAAHAClgg7H330kRITE/XXv/5VGRkZaty4seLj43XixAlHlwYAABzMEmFn6tSp6t+/v5566ik1aNBAs2fPlre3t+bMmePo0gAAgIO5OrqA8iosLFR6erpGjRpltrm4uCguLk5bt2697DYFBQUqKCgw3+fk5EiScnNz7V5fScFZu/eJssu1GY4uAb9XAcedI3CsOxeOdSdTQcf5r7/bhvHHf983fdj573//q+LiYoWEhJRqDwkJ0Q8//HDZbZKSkjR27NhL2iMiIiqkRjiPAEcXgEtN5G8F9sd/VU6mgo/zM2fOKCDgyp9x04edshg1apQSExPN9yUlJTp16pSCg4Nls9kcWBkqUm5uriIiInTs2DH5+/s7uhwAFYRj/dZhGIbOnDmj8PDwP1zvpg87t912mypVqqSsrKxS7VlZWQoNDb3sNh4eHvLw8CjVFhgYWFElwsn4+/vzP0DgFsCxfmv4oxGdX930E5Td3d3VvHlzJScnm20lJSVKTk5WdHS0AysDAADO4KYf2ZGkxMRE9enTRy1atFDLli01ffp05efn66mnnnJ0aQAAwMEsEXYeffRR/ec//9GYMWOUmZmpJk2aaOXKlZdMWsatzcPDQ3/9618vOYUJwFo41vF7NuNq12sBAADcxG76OTsAAAB/hLADAAAsjbADAAAsjbADh7HZbFq+fLkk6fDhw7LZbNq5c+c1bdu3b1917969wmoDgPKoWbOmpk+f7ugy8H8IO6gQmZmZev7551WrVi15eHgoIiJC3bp1K3U/pN+KiIjQ8ePHdeedd1ZYTb8NV79FcALso2/fvrLZbLLZbHJ3d1ft2rU1btw4XbhwoVz9rlu3TjabTdnZ2fYpFLccS1x6Dudy+PBhtW7dWoGBgXrzzTfVqFEjFRUVadWqVRo0aNBln1lWqVKlK97xGsDNo1OnTpo7d64KCgr01VdfadCgQXJzcyv1sGZHKiwslLu7u6PLwA3GyA7sbuDAgbLZbPrmm2/Us2dP3XHHHWrYsKESExO1bdu2y25zudNYe/bsUdeuXeXv7y8/Pz+1bdtWBw8evOz2aWlpqlKliiZNmlTu+gsKCjRkyBBVrVpVnp6eatOmjdLS0szl8+bNu+TxIsuXLy/1XLVdu3YpNjZWfn5+8vf3V/PmzbV9+3Zz+aZNm9S2bVt5eXkpIiJCQ4YMUX5+frlrBxzNw8NDoaGhqlGjhp577jnFxcXp888/1+nTp9W7d29VrlxZ3t7e6ty5sw4cOGBud+TIEXXr1k2VK1eWj4+PGjZsqK+++kqHDx9WbGysJKly5cqy2Wzq27evpIsPf+zVq5d8fHwUFhamadOm6b777tPQoUPNfmvWrKnx48erd+/e8vf314ABAyRd/Rh85513VKdOHXl6eiokJEQPPfSQuSw/P1+9e/eWr6+vwsLCNGXKlEs+F86FsAO7OnXqlFauXKlBgwbJx8fnkuXX+gyyn3/+We3atZOHh4dSUlKUnp6up59++rLD4SkpKerYsaMmTJigESNGlHcXNHz4cH366aeaP3++MjIyVLt2bcXHx+vUqVPX3EevXr10++23Ky0tTenp6Ro5cqTc3NwkSQcPHlSnTp3Us2dPffvtt/roo4+0adMmDR48uNy1A87Gy8tLhYWF6tu3r7Zv367PP/9cW7dulWEYeuCBB1RUVCRJGjRokAoKCrRhwwZ99913mjRpknx9fRUREaFPP/1UkrRv3z4dP35cb7/9tqSLd8/fvHmzPv/8c61Zs0YbN25URkbGJTW89dZbaty4sXbs2KHRo0df9Rjcvn27hgwZonHjxmnfvn1auXKl2rVrZ/Y3bNgwrV+/Xp999plWr16tdevWXfZz4UQMwI5SU1MNScbSpUuvuq4kY9myZYZhGMahQ4cMScaOHTsMwzCMUaNGGZGRkUZhYeFlt+3Tp4+RkJBgLF261PD19TUWL158TZ/n6elp+Pj4lHq5uroaCQkJhmEYRl5enuHm5mYsXLjQ3K6wsNAIDw83Jk+ebBiGYcydO9cICAgo1feyZcuM3x5Ofn5+xrx58y5bR79+/YwBAwaUatu4caPh4uJinDt37qr7ATirX49LwzCMkpISY82aNYaHh4fRvXt3Q5KxefNmc93//ve/hpeXl/Hxxx8bhmEYjRo1Ml577bXL9rt27VpDknH69GmzLTc313BzczOWLFlitmVnZxve3t7GCy+8YLbVqFHD6N69e6n+rnYMfvrpp4a/v7+Rm5t7SS1nzpwx3N3dzboNwzBOnjxpeHl5XfK506ZNu+z+4MZjzg7syrDTDbl37typtm3bmqMhl5OamqoVK1bok08+ueYJxtOmTVNcXFypthEjRqi4uFjSxVGXoqIitW7d2lzu5uamli1bau/evddcf2Jiov7yl7/ogw8+UFxcnB5++GFFRUVJuniK69tvv9XChQvN9Q3DUElJiQ4dOqT69etf8+cAzmbFihXy9fVVUVGRSkpK9Oc//1k9evTQihUr1KpVK3O94OBg1a1b1zyuhgwZoueee06rV69WXFycevbsqbvuuuuKn/PTTz+pqKhILVu2NNsCAgJUt27dS9Zt0aJFqfdXOwY7duyoGjVqqFatWurUqZM6deqkBx98UN7e3jp48KAKCwtL7UtQUNBlPxfOg9NYsKs6derIZrNddhLy9fDy8rrqOlFRUapXr57mzJljDoVfTWhoqGrXrl3q5efnd121ubi4XBLqfv/5r732mvbs2aMuXbooJSVFDRo00LJlyyRJeXl5euaZZ7Rz507ztWvXLh04cMAMRMDNKjY2Vjt37tSBAwd07tw5zZ8/v9R8tiv5y1/+op9++klPPvmkvvvuO7Vo0UIzZ860S02/P6V+tWPQz89PGRkZ+vDDDxUWFqYxY8aocePGXA12EyPswK6CgoIUHx+vWbNmXXbC7bX+z+Kuu+7Sxo0b/zDE3HbbbUpJSdGPP/6oRx555JoDzx+JioqSu7u7Nm/ebLYVFRUpLS1NDRo0kCRVqVJFZ86cKbV/l7s/0B133KEXX3xRq1evVo8ePTR37lxJUrNmzfT9999fErpq167NVSK46fn4+Kh27dqqXr26XF0vnjyoX7++Lly4oNTUVHO9kydPat++feZxJV28BcWzzz6rpUuX6qWXXtL7778vSeZx8esIrCTVqlVLbm5upS4eyMnJ0f79+69a47Ucg66uroqLi9PkyZP17bff6vDhw0pJSVFUVJTc3NxK7cvp06ev6XPhOIQd2N2sWbNUXFysli1b6tNPP9WBAwe0d+9ezZgxQ9HR0dfUx+DBg5Wbm6vHHntM27dv14EDB/TBBx9o3759pdarWrWqUlJS9MMPP+jxxx8v9/08fHx89Nxzz2nYsGFauXKlvv/+e/Xv319nz55Vv379JEmtWrWSt7e3XnnlFR08eFCLFi3SvHnzzD7OnTunwYMHa926dTpy5Ig2b96stLQ08/TUiBEjtGXLFg0ePNj8F/Bnn33GBGVYVp06dZSQkKD+/ftr06ZN2rVrl5544glVq1ZNCQkJkqShQ4dq1apVOnTokDIyMrR27VrzmKlRo4ZsNptWrFih//znP8rLy5Ofn5/69OmjYcOGae3atdqzZ4/69esnFxeXq44kXe0YXLFihWbMmKGdO3fqyJEjWrBggUpKSlS3bl35+vqqX79+GjZsmFJSUrR792717dtXLi78nDoz/nZgd7Vq1VJGRoZiY2P10ksv6c4771THjh2VnJysd99995r6CA4OVkpKivLy8hQTE6PmzZvr/fffv+wcntDQUKWkpOi7775Tr169Sv3rrywmTpyonj176sknn1SzZs30448/atWqVapcubKki6NX//rXv/TVV1+pUaNG+vDDD/Xaa6+Z21eqVEknT55U7969dccdd+iRRx5R586dNXbsWEkXR63Wr1+v/fv3q23btmratKnGjBmj8PDwctUNOLO5c+eqefPm6tq1q6Kjo2UYhr766ivzmC4uLtagQYNUv359derUSXfccYfeeecdSVK1atU0duxYjRw5UiEhIWYomTp1qqKjo9W1a1fFxcWpdevWql+/vjw9Pf+wlqsdg4GBgVq6dKnat2+v+vXra/bs2frwww/VsGFDSdKbb76ptm3bqlu3boqLi1ObNm3UvHnzivrqYAc2w14zSgEAcKD8/HxVq1ZNU6ZMMUdib5T77rtPTZo04RERToqrsQAAN6UdO3bohx9+UMuWLZWTk6Nx48ZJknlqDPgVYQcAcNN66623tG/fPrm7u6t58+bauHGjbrvtNkeXBSfDaSwAAGBpTFAGAACWRtgBAACWRtgBAACWRtgBAACWRtgBAACWRtgBYDnr1q2TzWbjwY0AJBF2AFSAvn37ymazyWazyc3NTSEhIerYsaPmzJmjkpKSa+5n3rx5CgwMrLhCAdwSCDsAKkSnTp10/PhxHT58WF9//bViY2P1wgsvqGvXruV+YCsAXA/CDoAK4eHhodDQUFWrVk3NmjXTK6+8os8++0xff/21+ZT4qVOnqlGjRvLx8VFERIQGDhyovLw8SRdPRT311FPKyckxR4l+feDqBx98oBYtWsjPz0+hoaH685//rBMnTlxSw+bNm3XXXXfJ09NT99xzj3bv3m0uO3nypB5//HFVq1ZN3t7e5kNdf+uTTz5Ro0aN5OXlpeDgYMXFxSk/P99c/o9//MN88GS9evXMB1dKUmFhoQYPHqywsDB5enqqRo0aSkpKstfXC+A6EHYA3DDt27dX48aNtXTpUkmSi4uLZsyYoT179mj+/PlKSUnR8OHDJUn33nuvpk+fLn9/fx0/flzHjx/Xyy+/LEkqKirS+PHjtWvXLi1fvlyHDx9W3759L/m8YcOGacqUKUpLS1OVKlXUrVs3FRUVSZLOnz+v5s2b68svv9Tu3bs1YMAAPfnkk/rmm28kScePH9fjjz+up59+Wnv37tW6devUo0cP/XrT+YULF2rMmDGaMGGC9u7dqzfeeEOjR4/W/PnzJUkzZszQ559/ro8//lj79u3TwoULVbNmzYr8egFciQEAdtanTx8jISHhssseffRRo379+pddtmTJEiM4ONh8P3fuXCMgIOCqn5eWlmZIMs6cOWMYhmGsXbvWkGQsXrzYXOfkyZOGl5eX8dFHH12xny5duhgvvfSSYRiGkZ6ebkgyDh8+fNl1o6KijEWLFpVqGz9+vBEdHW0YhmE8//zzRvv27Y2SkpKr1g+gYjGyA+CGMgxDNptNkvTvf/9bHTp0ULVq1eTn56cnn3xSJ0+e1NmzZ/+wj/T0dHXr1k3Vq1eXn5+fYmJiJElHjx4ttV50dLT556CgINWtW1d79+6VJBUXF2v8+PFq1KiRgoKC5Ovrq1WrVpl9NG7cWB06dFCjRo308MMP6/3339fp06clSfn5+Tp48KD69esnX19f8/X666/r4MGDki5O0t65c6fq1q2rIUOGaPXq1Xb49gCUBWEHwA21d+9eRUZG6vDhw+ratavuuusuffrpp0pPT9esWbMkXZzvciX5+fmKj4+Xv7+/Fi5cqLS0NC1btuyq2/3em2++qbffflsjRozQ2rVrtXPnTsXHx5t9VKpUSWvWrNHXX3+tBg0aaObMmapbt64OHTpkzit6//33tXPnTvO1e/dubdu2TZLUrFkzHTp0SOPHj9e5c+f0yCOP6KGHHirTdwagfFwdXQCAW0dKSoq+++47vfjii0pPT1dJSYmmTJkiF5eL/+76+OOPS63v7u6u4uLiUm0//PCDTp48qYkTJyoiIkKStH379st+3rZt21S9enVJ0unTp7V//37Vr19f0sXJywkJCXriiSckSSUlJdq/f78aNGhgbm+z2dS6dWu1bt1aY8aMUY0aNbRs2TIlJiYqPDxcP/30k3r16nXF/fX399ejjz6qRx99VA899JA6deqkU6dOKSgo6Hq+NgDlRNgBUCEKCgqUmZmp4uJiZWVlaeXKlUpKSlLXrl3Vu3dv7d69W0VFRZo5c6a6deumzZs3a/bs2aX6qFmzpvLy8pScnKzGjRvL29tb1atXl7u7u2bOnKlnn31Wu3fv1vjx4y9bw7hx4xQcHKyQkBD9z//8j2677TZ1795dklSnTh198skn2rJliypXrqypU6cqKyvLDDupqalKTk7W/fffr6pVqyo1NVX/+c9/zLA0duxYDRkyRAEBAerUqZMKCgq0fft2nT59WomJiZo6darCwsLUtGlTubi4aMmSJQoNDeW+QYAjOHrSEADr6dOnjyHJkGS4uroaVapUMeLi4ow5c+YYxcXF5npTp041wsLCDC8vLyM+Pt5YsGCBIck4ffq0uc6zzz5rBAcHG5KMv/71r4ZhGMaiRYuMmjVrGh4eHkZ0dLTx+eefG5KMHTt2GIbx/xOUv/jiC6Nhw4aGu7u70bJlS2PXrl1mvydPnjQSEhIMX19fo2rVqsarr75q9O7d25xY/f333xvx8fFGlSpVDA8PD+OOO+4wZs6cWWo/Fy5caDRp0sRwd3c3KleubLRr185YunSpYRiG8fe//91o0qSJ4ePjY/j7+xsdOnQwMjIy7P9lA7gqm2H833WUAAAAFsQEZQAAYGmEHQAAYGmEHQAAYGmEHQAAYGmEHQAAYGmEHQAAYGmEHQAAYGmEHQAAYGmEHQAAYGmEHQAAYGmEHQAAYGn/C0ezOk56ksJiAAAAAElFTkSuQmCC", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "x = np.arange(2) # change here\n", "width = 0.40\n", "y1 = [cHouse_read_execution_time, psql_read_execution_time] # change here\n", "y2 = [cHouse_write_execution_time, psql_write_execution_time] # change here\n", "plt.bar(x - 0.2, y1, width)\n", "plt.bar(x + 0.2, y2, width)\n", "plt.xticks(x, [\"Click House\", \"Postgresql\"])\n", "plt.xlabel(\"Databases\")\n", "plt.ylabel(\"Seconds\")\n", "plt.legend([\"Read\", \"Write\"]) # ver\n", "plt.show()" ] }, { "cell_type": "code", "execution_count": null, "id": "cac1c82a-bcab-47b1-b302-77d1341e5304", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.11" }, "widgets": { "application/vnd.jupyter.widget-state+json": { "state": {}, "version_major": 2, "version_minor": 0 } } }, "nbformat": 4, "nbformat_minor": 5 }