patg.net

Clickhouse S3

Tue, 08 Oct 2024 00:00:00 +0000

Clickhouse Database Imports data from S3!

I have been trying out the Clickhouse Database for a few weeks now and really impressed. For those who don’t know, ClickHouse is a high-performance, columnar-oriented database management system designed primarily for real-time analytics. It was developed by Yandex and is known for its ability to process very large volumes of data quickly while using minimal system resources. ClickHouse excels in handling analytical queries with high throughput and low latency, making it a popular choice for big data, log analysis, time-series data, and online analytical processing (OLAP) tasks. Companies such as Altinity, Clickhouse, (first two who offer SaaS around Clickhouse) Craigslist, Dashdive, Deutsche Bank, and many others. The Github repo is a good place to get started.

In using it, much of my background with MySQL and PostgreSQL has helped me, plus, it actually has interfaces for MySQL and PostgreSQL clients!

S3 Functionality

One feature I find really interesting and useful is the ability to query data from AWS S3.

Setup

Create a bucket:

aws s3 mb s3://clickhouse-patg

Copy a gzipped TSV file with data (headers in this one as well)

aws s3 cp data.tsv.gz s3://clickhouse-patg/

Set the server preferences with a file /etc/clickhouse-server/config.d/s3.xml:

<clickhouse>
    <s3>
        <endpoint-name>
            <endpoint>https://clickhouse-patg.s3.amazonaws.com</endpoint>
            <access_key_id>AKredacted</access_key_id>
            <secret_access_key>xyzredacted</secret_access_key>
            <!-- <use_environment_credentials>false</use_environment_credentials> -->
            <!-- <header>Authorization: Bearer SOME-TOKEN</header> -->
        </endpoint-name>
    </s3>
</clickhouse>

Restart the server (Ubuntu 24.04)

systemctl restart clickhouse-server.service

Query the table!

clickhouse-client --password # uses 'default' user and password set on install

myhost.internal :) select * from s3('https://clickhouse-patg.s3.amazonaws.com/data.tsv.gz','TabSeparatedRaw') limit 10;

SELECT *
FROM s3('https://clickhouse-patg.s3.amazonaws.com/data.tsv.gz', 'TabSeparatedRaw')
LIMIT 10

Query id: 3b587b74-4803-48e1-9492-13042794de55

   ┌─id	stringval	digit────────────────────┐
1. │ 0	nvlpuJWZ78rM4Ma9	3.5153584288539355 │
2. │ 1	4NY8I0R1DAjVGGHO	26.154132477041742 │
3. │ 2	9rof1FJDFg3CvAb6	60.927590847441266 │
4. │ 3	Nx1jGSRf8VOh5TSS	84.07472488753888  │
5. │ 4	PWriUaORQc32mbcP	78.96101702751513  │
6. │ 5	DxJmb0ASVZiGIeef	17.171794187401467 │
7. │ 6	fy5peZZeJqwMpt5Q	38.114516479322496 │
8. │ 7	A0LeX5v7WghiFbva	30.961514875964678 │
9. │ 8	Bqi2IInBlisHSYeS	35.935753912623916 │
   └──────────────────────────────────────────────────┘
    ┌─id	stringval	digit────────────────────┐
10. │ 9	KNHLNmxFRMAPKRMr	52.692043323314586 │
    └──────────────────────────────────────────────────┘

10 rows in set. Elapsed: 1.111 sec. 

infra.cs342cloud.internal :) 

Now, insert the data from S3 into a table on the server with this data:

INSERT INTO testdata (id, stringval, digit) SELECT *
FROM s3('https://clickhouse-patg.s3.amazonaws.com/data.tsv.gz', 'TabSeparatedRaw')

Query id: 72031373-f4bd-4fab-9164-9f5df6b30036

Ok.

0 rows in set. Elapsed: 2.718 sec. Processed 2.00 million rows, 66.00 MB (735.87 thousand rows/s., 24.28 MB/s.)
Peak memory usage: 123.58 MiB.

myhost.internal :) select * from testdata limit 10;

SELECT *
FROM testdata
LIMIT 10

Query id: 74ca3a98-0f06-4e01-9bdc-812d9b8048dd

    ┌─id─┬─stringval────────┬─────digit─┐
 1. │  0 │ nvlpuJWZ78rM4Ma9 │ 3.5153584 │
 2. │  1 │ 4NY8I0R1DAjVGGHO │ 26.154133 │
 3. │  2 │ 9rof1FJDFg3CvAb6 │  60.92759 │
 4. │  3 │ Nx1jGSRf8VOh5TSS │  84.07472 │
 5. │  4 │ PWriUaORQc32mbcP │  78.96101 │
 6. │  5 │ DxJmb0ASVZiGIeef │ 17.171795 │
 7. │  6 │ fy5peZZeJqwMpt5Q │ 38.114517 │
 8. │  7 │ A0LeX5v7WghiFbva │ 30.961515 │
 9. │  8 │ Bqi2IInBlisHSYeS │ 35.935753 │
10. │  9 │ KNHLNmxFRMAPKRMr │ 52.692043 │
    └────┴──────────────────┴───────────┘

10 rows in set. Elapsed: 0.006 sec. 

Very cool indeed!

Summary

Clickhouse is an amazing database thus far, and as shown above, one of the many things it can do is utilize data in S3, in this example, query or insert.

Stay tuned for more!

Godaddy Terraform

Tue, 16 Apr 2024 00:00:00 +0000

Using Terraform With Godaddy

So, to start off this post, I haven’t posted on my blog for, a while. I had just been super busy both at Oracle Cloud and then in the role of CTO and CIO at Parler, which after a day of work, and having kids at the premium age for interaction, and I suppose fatigue from being in this industry since 1993, and really technical field going back to the day I went into the Navy in 1986, you get the point!

In any case, one of the things I thought about when I was climbing Aconcagua and spending time with Peter Zaitsev, who is a tremendous person both professionally as well as in general, I want to get back to contributing to the open-source and technical community. I wrote two books and can put out material quickly when I get down to it, so here goes the resumption of that endevour.

Why this post?

One of the things I’ve been needing to do, especially to keep my skills up to date, is to refresh some of my knowledge on using cloud-specific technologies. For the last three years, I’ve been working in environments where we had to build our own infrastucture and not having the luxury of using tools like Terraform, or using AWS, though I have written books of Ansible to manage databases (MySQL and Postgres) as well as various varieties of do-it-yourself Kubernetes clusters.

The first task I wanted to do was to build my own EKS Kubernetes cluster and use a domain I have, which happens to be “kubernetes-cluster.com” (I obtained it in early days). To do this, I realized I needed to modify my NS records for that domain, which the registrar is Godaddy, and I want to use Route 53 to manage it and in turn, use External DNS with Kubernetes.

I thought “OK, I have two things I need to do that can be done with one project. Modify the kubernetes-cluster.com’s NS records and do something useful with Terraform. And for sure, there was a provider for Godaddy!

The Godaddy Provider for Terraform

The Goddady Provider for Terraform is incredibly simple to use. For the sake of getting this blog post done, I’m just going to show you the main.tf file that I created to change the nameservers for kubernetes-cluster.com to use AWS’s Route53 DNS. I had never used this provider before, so there is no existing state, but simply specifying the nameservers for Route53, it replaced the old nameservers with what I wanted.

main.tf:

terraform {
  required_providers {
    godaddy = {
      source = "n3integration/godaddy"
      version = "1.9.1"
    }
  }
}

provider "godaddy" {
  # Configuration options
}

provider "aws" {
  region  = "us-west-2"
  profile = "default"

  backend s3 {
    bucket = "tf-godaddy-state"
    key    = "domains/kubernetes_cluster_com"
    region = "us-east-1"
  }
}
resource "godaddy_domain_record" "kubernetes_cluster_com" {
  domain   = "kubernetes-cluster.com"

  // your Godaddy customer ID obviously
  customer = "123456788888888"

  // AWS nameservers specified in Route53
  nameservers = ["ns-526.awsdns-01.net","ns-1521.awsdns-62.org","ns-1975.awsdns-54.co.uk","ns-5.awsdns-00.com"]

}

First run terraform init (I use an alias tf set in my ~/.bashrc)

$ tf init

Initializing the backend...

Initializing provider plugins...
- Reusing previous version of n3integration/godaddy from the dependency lock file
- Reusing previous version of hashicorp/aws from the dependency lock file
- Using previously-installed n3integration/godaddy v1.9.1
- Using previously-installed hashicorp/aws v5.45.0

Terraform has been successfully initialized!

Then of course, run terraform plan:

tf plan

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following
symbols:
  + create

Terraform will perform the following actions:

  # godaddy_domain_record.kubernetes_cluster_com will be created
  + resource "godaddy_domain_record" "kubernetes_cluster_com" {
      + customer    = "12345678"
      + domain      = "kubernetes-cluster.com"
      + id          = (known after apply)
      + nameservers = [
          + "ns-526.awsdns-01.net",
          + "ns-1521.awsdns-62.org",
          + "ns-1975.awsdns-54.co.uk",
          + "ns-5.awsdns-00.com",
        ]
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Then of course run terraform apply. I won’t include that output for the purpose of brevity, and because it’s what you’d expect.

Import to note: after I changed the nameservers from Goddady providing DNS, this makes it so that Goddady no longer has a zone file for kubernetes-cluster.com and is only serving as a registrar, so I can’t use the Terraform provider if I want to make changes to the domain and will need to use the Route53 terraform provider.

I also used this provider for adding a cname for patg.net:

resource "godaddy_domain_record" "patg-net" {
  domain   = "patg.net"

  // required if provider key does not belong to customer
  customer = "12345678"

  record {
    name = "blog"
    type = "CNAME"
    data = "patg.net"
    ttl = 3600
  }

}

Again, quite simple, using the same steps (init, plan, apply) and being only adding a CNAME, I can continue to use this provider.

Have a great day, and thanks for reading my post!

Get Password

Tue, 16 Apr 2024 00:00:00 +0000

Here is a nice one-liner to get the password and username set in a kubernetes secret for the Grafana UI after installing Prometheus:

kubectl get secret patg-prometheus-stack-grafana -o json|jq '.data."admin-password"'|sed -e 's/"//g'|base64 -D

Running XtraBackup on Windows using WSL

Sun, 03 Sep 2017 12:00:00 +0000

Backing up MySQL on Windows: options?

I had a recurring consultation this summer with a client who is running a stand-alone MySQL database with a ton of data and wanted to simply have a backup system for that data. The client is very risk-averse and didn’t want anything too experimental. I knew for a huge database, I didn’t want to run mysqldump on the database on a regular basis if I could avoid it and always prefer using XtraBackup.

I haven’t worked on Windows in years, and had assumed there were binaries for XtraBackup, but as with so many things with Windows, there are a million assumptions I had made prior that I have learned the hard way throughout the summer.

I did find Vadim’s excellent article using a Docker container to run a container with XtraBackup, and bind mounts of both the datadir and the backup directory so the container can run a backup and have the backup show up on the host where specified in the arguments. This is certainly the way I gravitate to as I find containers a great solution to so many problems. However, again, I mentioned the client wanted something that isn’t perceived to be to experimental, so I wanted to explore using something that was “part” of Windows.

WSL: Windows Subsystem for Linux

Microsoft certainly is a different company than it used to be. Microsoft has really put a lot of work into being part of Open Source, VSCode, the container ecosystem, Kubernetes, Open-Sourcing PowerShell, etc. Also one can consider the context of Microsoft being serious about attracting developers to their platform. In addition to these other efforts, my friend James Hancock, an expert in all things Windows, told me about WSL, the Windows Subsystem for Linux.

I had always known that Windows had this idea of a base kernel with subsytems, both a Windows subsystem as well as a POSIX subsystem. I hadn’t used Windows much in years, so I didn’t know the status of what you could do with Windows in current times.

WSL was an effort inside of Microsoft researching real time translation of Linux system calls into Windows OS system calls, and derived from their work in working with Android apps to run on Windows 10 Mobile.

What is WSL really?

Throughout the years, there have been a number of ways to run “unix” or posix commands on Windows with some sort of bash shell for those who are Linux people to get by when working on Windows. WSL is much more than that though.

WSL is not CygWin nor MinGW.

Those are great tools, but not a complete subsystem like WSL that integrates into the Windows OS

WSL is not a container nor VM running Linux

WSL is part of the OS, a subsystem just like Windows itself and equal footing and not just an accessory program

WSL is currently like having Ubuntu on Windows

When you run WSL, you have an Ubuntu installation you can do all the same things for the most part you would if you were running on a VM or bare metal.

On WSL, there exists a regular Linux file system with Windows fixed drives mounted as /mnt/<drive letter> so that you have access to you Windows files as well.

It’s also an excellent development platform as now one can develop both for Windows or Linux.

Using WSL

It’s part of the Windows OS, so it just needs to be set up to run.

Basically, it can be enabled through enabling developer mode and turning on the WSL feature in Settings->Update & Security, as detailed in this article

Or using a simple PowerShell command as detailed in this article

Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux

It can also be run as a specific user if desired

LxRun.exe /setdefaultuser <new_name>

What does WSL mean for XtraBackup?

Since there aren’t any Windows binaries XtraBackup, one has to either use a Docker container or WSL.

For WSL, it is Ubuntu. So, to install XtraBackup for WSL, install XtraBackup as it would be installed on Ubuntu! Yes, that easy.

Setting up XtraBackup on WSL

The current Ubuntu version for WSL is xenial, so that is the version, using lsb-release to get the right package version.The lsb package will need to be installed to make it work.

apt-get update
apt-get install lsb
wget https://repo.percona.com/apt/percona-release_0.1-4.$(lsb_release -sc)_all.deb
sudo dpkg -i percona-release_0.1-4.$(lsb_release -sc)_all.deb
sudo apt-get update
sudo apt-get install percona-xtrabackup-24

The MySQL version on the server was installed from Bitnami’s WAMP package which installs MySQL in C:\Bitnami\wampstack-7.1.7-0\mysql\data, but we modified the datadir to be in C:\mysqldata, which is accessible as /mnt/c/mysqldata in WSL.

The first thing is to decide where to back up data.

IMPORTANT: the current version of Windows, WSL cannot access USB drives, so you if data needs to be on a detachable USB drive, it’ll need to be copied from the attached storage to the USB drive. The current insider release does address this and soon this problem won’t exist on regular releases.

For a backup directory, In our case, we chose C:\backups for simplicity.

mkdir /mnt/c/backups

So, with a running MySQL instance and a place to create backups, XtraBackup can be run via innobackupex

innobackupex --datadir=/mnt/c/mysqldata /mnt/c/backups
innobackupex --apply-log /mnt/c/backups/2017-09-xxx

And That’s it! Now you can have yet another way to back up MySQL on Windows!

HAProxy Tip

Fri, 04 Aug 2017 12:00:00 +0000

Question: how do I have HAProxy redirect to another site and preserve the full URL?

I’ve been thinking: sometimes I feel that in order to post to my blog, it has to be significant. This runs contrary to the whole idea of “release early, release often” ideal in terms of documentation. Why not just post findings when they’re fresh in my mind? Here goes.

Today, I had a client who needed one site redirect another site and retain the full URL on the web front-end, which uses HAProxy. The existing redirects would do this for other sites, but would redirect to the front page and if whatever the full URL was that was acccessed was lost.

It was a pretty simple problem to solve, but not a lot of examples out there that were more complex.

First things first

When you test front-end proxy changes, disable your browswer’s cache! In Chrome, this is done with Dev tools->Network->check “Disable cache”. Other browswers can also have their cache disabled. Yes, I know, that seems obvious, but shift reload doesn’t work well nor is convenient when you cut-n-paste a URL from some other application or email to test your changes or have to explicitely clear it.

Also, have a test proxy that you spin up to make changes on and use /etc/hosts to have it respond as the front-end web proxy.

Original rules in HAProxy

The original rule went to a subdirectory on the main site, but this needed to be changed to have it go to a completely different site

acl is_somesite hdr(host) -i somesite.com
redirect code 301 location http://www.mainsite.com/somesite/ if is_somesite

It now needed instead to go from http://somesite.com/path?query_string to http://anothersite.com/path?query_string. The former rule didn’t work as:

redirect code 301 location http://www.anothersite.com if is_somesite

And would obviously go, regardless of the full URI, to http://www.anothersite.com

After searching, reading the manual, and specifically for the version being used, this did the trick:

acl is_somesite hdr_end(host) -i somesite.com
http-request redirect code 301 location http://www.anothersite.com%[capture.req.uri] if is_somesite

Note the use of hdr_end(host). The reason for this is it takes care of any subdomain and the main domain. Also, %[capture.req.uri] is the real magic for ensuring the full URL is carried over.

For the most recent version of HAProxy (7.0), it would be req.uri versus capture.req.uri

Summary

So, there it is. Hopefully this will help others save time.

AMI creation - what is your process?

Mon, 24 Jul 2017 12:00:00 +0000

What is your process for creating AMIs?

I’ve had a discussion with a colleague about the process of creating AMIs, or more specifically, dealing with instances that need some sort of state captured. AWS offers you the ability to create an image off of an instance. We were pondering in our discussion how that can be possible with the situation of how can AWS ensure that the state of what is being made into an image is what you expect it to be? Open file handles, things in memory needing to be flushed to disk. Is this feature of EC2 something that should even be used for an authoritative state you expect your image to be?

In our case, it was an instance that was on a host that had problems and needed to be moved off and the underlying host become unresponsive (according to AWS) and the instance couldn’t even be shut down.

The basic way to go about it would be to build a new instance, provision it the way you want it (Ansible, Chef, manual configuration) and once complete, stop the instance, and make an image from that.

What is your process? And those of you intimate with the details of EC2, how fool-proof is the ‘create an image’ functionality?

Percona Chart now in Kubernetes Charts

Sat, 15 Apr 2017 12:00:00 +0000

Percona Helm Chart

I’m happy to see that my pull request was approved. Pretty simply, it was a Percona Chart. We needed the setup at HPE for our cluster and it was a good excuse to put together a chart, so there ya go!

To use it, simply run helm install --name my-release stable/percona

What is a Helm chart?

Well, for those that don’t know, some prerequisite questions:

What is Helm?

Helm is a package manager for Kubernetes, these packages called “charts”

What is Kubernetes?

Kubernetes is an open-sourced system for managing containerized applications across multiple hosts

What are containers?

Containers are concept of technology in an operating system that isolation such that you can package an application and ship it anywhere and have it run the same exact way.

Pokémon GO scaling with Couchbase

Wed, 02 Nov 2016 12:00:00 +0000

Interesting article on how Pokémon GO is scaling profile services with Couchbase & NoSQL

Terraform setup repository for AWS

Wed, 21 Sep 2016 12:00:00 +0000

Just a quick post for a project I put together using Terraform to build out ready-to-use networking on aws

What it does in a nutshell

This project basically builds out a VPC with 4 subnets (one public, three private) that is all ready to use to build out your AWS environment using Terraform.

It’s really simple to use. You just need to edit the variables.tf to suit your AWS account credentials as well as whatever else you want to adjust.

Sanity check with terraform plan and then launch with terraform apply.

That’s it!

Add more hosts, run whatever you need. All the networking (routing, gateways, security groups, etc) is ready to use.

Running Kubernetes with Fleet on CoreOS

Tue, 12 Jul 2016 12:00:00 +0000

I’m pleased to announce publicly available Fleet Unit Files for launching a multi-node Kubernetes cluster on CoreOS, kubernetes-cluster-fleet

Background

I have run Kubernetes in any number of ways, particularly using Ansible to set it up, but was recently wanting to see if it was possible to run Kubernetes completely using Fleet unit files on CoreoS– across multiple machines, something I haven’t found a solution for thus far other than some unit files that run Kubernetes on a single-machine. After a lot of trial and error and bash duct tape, it is indeed possible, hence my repository https://github.com/CaptTofu/kubernetes-cluster-fleet

Basic idea

When one runs Kubernetes, the API server is run, and it has in etcd an entry for the IP address it’s running on found in the entry /registry/services/endpoints/default/kubernetes. The unit files for every other Kubernetes component use this entry to set the value of what the API server ought to be.
There are also unit files that set up the environment and set the host/IP entries in /etc/hosts of every server to make it easier to deploy.

Usage

Review the fine documentation on the project page for how to run this! It is pretty simple and straightforward.