# Today I learned!
Why this space?
This space is to jot down all the learning and to document the things I have learned today. The today I learned space is being inspired by Simon Willison(creator of Django) who maintains a TIL on it's own.
{% embed url="" %}
I used to write the things in my personal blog, but I feel this requires a place on it's own and should be separated from my blog. Thus this space was created.
Other interesting TILs by awesome folks:
- KraceKumar
- Vincent D. Warmerdam
Simon Willison
There is an awesome gitbook by Santhost Thottingal which inspired me to create it with this platform:\
\
{% embed url="" %}
# Deployment
# Rolling, Canary, Blue-green deployments
**Rolling Deployment**
Deployments support updating images to a new version through a rolling update mechanism. When a Deployment is updated with a new version, it creates a new ReplicaSet and slowly increases the number of replicas in the new ReplicaSet as it decreases the replicas in the old ReplicaSet.
In kubernetees, it's done by editing deployment with new version and updating image with latest version of deployment
It possible to pause, resume, rollback new changes with this deployment.
**Canary Deployment**
A canary deployment consists of a separate deployment with your new version and a service that targets both your normal, stable deployment as well as your canary deployment. Usually needs a new track version called canary in kubernetees. Also a few deployments are using new version initially.
### Blue-green deployments
Rolling updates are ideal because they allow you to deploy an application slowly with minimal overhead, minimal performance impact, and minimal downtime. There are instances where it is beneficial to modify the load balancers to point to that new version only after it has been fully deployed. In this case, blue-green deployments are the way to go.
Kubernetes achieves this by creating two separate deployments; one for the old "blue" version and one for the new "green" version. Use your existing `hello` deployment for the "blue" version. The deployments will be accessed via a Service which will act as the router. Once the new "green" version is up and running, you'll switch over to using that version by updating the Service.
**References:**
{% embed url="" %}
{% embed url="" %}
{% embed url="" %}
# Kubernetees Helm Charts
In simple terms, Helm is a package manager for Kubernetes. Helm is the K8s equivalent of yum or apt. Helm deploys charts, which you can think of as a packaged application. It is a collection of all your versioned, pre-configured application resources which can be deployed as one unit.
To fully grasp helm, there are 3 concepts we need to get familiar with:
* **Chart**: A package of pre-configured Kubernetes resources.
* **Release**: A specific instance of a chart which has been deployed to the cluster using Helm.
* **Repository**: A group of published charts which can be made available to others.
Seldon can be installed via Kustomize and Helm
# AI/ML
# SeldonIO
Seldon core converts your ML models (Tensorflow, Pytorch, H2o, etc.) or language wrappers (Python, Java, etc.) into production REST/GRPC microservices.
Seldon handles scaling to thousands of production machine learning models and provides advanced machine learning capabilities out of the box including Advanced Metrics, Request Logging, Explainers, Outlier Detectors, A/B Tests, Canaries and more.
#### Deploy your model using pre-packaged model servers[¶](https://docs.seldon.io/projects/seldon-core/en/latest/workflow/github-readme.html#deploy-your-model-using-pre-packaged-model-servers)
We provide optimized model servers for some of the most popular Deep Learning and Machine Learning frameworks that allow you to deploy your trained model binaries/weights without having to containerize or modify them.
**Send API requests to your deployed model**[**¶**](https://docs.seldon.io/projects/seldon-core/en/latest/workflow/github-readme.html#send-api-requests-to-your-deployed-model)
Every model deployed exposes a standardised User Interface to send requests using our OpenAPI schema.
This can be accessed through the endpoint `http:///seldon///api/v1.0/doc/` which will allow you to send requests directly through your browser
These are Seldon Core main components:
* Reusable and non-reusable [model servers](https://docs.seldon.io/projects/seldon-core/en/latest/workflow/overview.html#e2e-serving-with-model-servers)
* [Language Wrappers](https://docs.seldon.io/projects/seldon-core/en/latest/workflow/overview.html#language-wrappers) to containerise models
* [SeldonDeployment](https://docs.seldon.io/projects/seldon-core/en/latest/workflow/overview.html#seldondeployment-crd) CRD and [Seldon Core Operator](https://docs.seldon.io/projects/seldon-core/en/latest/workflow/overview.html#seldon-core-operator)
* [Service Orchestrator](https://docs.seldon.io/projects/seldon-core/en/latest/workflow/overview.html#service-orchestrator) for advanced inference graphs
as well as integration with third-party systems:
* Kubernetes Ingress integration with [Ambassador](https://www.getambassador.io) and [Istio](https://istio.io)
* [Metrics](https://docs.seldon.io/projects/seldon-core/en/latest/workflow/overview.html#metrics-with-prometheus) with [Prometheus](https://prometheus.io)
* [Tracing](https://docs.seldon.io/projects/seldon-core/en/latest/workflow/overview.html#distributed-tracing-with-jaeger) with [Jaeger](https://www.jaegertracing.io)
* [Endpoint Documentation](https://docs.seldon.io/projects/seldon-core/en/latest/workflow/overview.html#endpoint-documentation) with [OpenApi](https://swagger.io/docs/specification/about/)
To learn more about Seldon refer below links:
*
* docs.seldon.io
*
*
## What is difference b/w Seldon Core and MLServer?
Basically Seldon Core is the "orchestrator" that takes your model, deploys it on to a server, collects metrics and logs and creates services and routing over the top. MLServer is the "engine" that actually handles executions of that model during inference. Seldon Core uses MLServer as a runtime to deploy your models on.
Another big difference is seldon-io core uses Flask under it's backend (for non-parallelization).
While ML-server is build with FastAPI, so our APIs with seldon can work on top of it as well.
# Installing software in E2E cloud compute
Most of the instances in E2E cloud comes with Python, but not with pip. CUDA drivers are install, but pytorch or tensorflow is missing.\
```
apt-get install python3-pip
```
Check CUDA version:\
```
nvcc --version
```
Then install pytorch for cuda:
```
python3 -m pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
```
Verify the installation using:\
```
import torch
torch.cuda.is_available()
```
Install jupyter lab and run with no password:
```
pip install jupyterlab
jupyter lab --allow-root --ip='*' --NotebookApp.token='' --NotebookApp.password=''
```
Then expose URL with ngrok: `ngrok http 8888`\
# Watching nvidia-smi
One of the most common things as a datascientist we do is watching nvidia-smi , to look at our GPU utilization, memory etc.
Simply running nvidia-smi will give the GPU and memory utilization at a particular time only.
In order to run it across a period of time, so you don't want to repeat yourself you can use the command:
```
nvidia-smi -l 1
watch -n0.1 nvidia-smi
```
Another command to check utilization, which was recommended by Jeremy Howard is:
```
nvidia-smi dmon
```
# How does github copilot works?
Parth Thakkar, had created an interesting analysis on how github copilot works:
{% embed url="" %}
* [Copilot Explorer](https://thakkarparth007.github.io/copilot-explorer) (2022)
In this project, I delved into the internals of Github Copilot. I [wrote](https://github.com/thakkarparth007/copilot-inspector) a tool to decompile Copilot's VSCode extension, and identified key insights regarding prompt construction logic, post processing of model's output and telemetry collection (*after sifting through over 400 obfuscated modules!*). I also built a [code browser](https://thakkarparth007.github.io/copilot-explorer/codeviz/templates/code-viz.html) to intuitively explore the codebase. I wrote a [blog post](https://thakkarparth007.github.io/copilot-explorer/posts/copilot-internals) about it that got a [little](https://twitter.com/karpathy/status/1608895189078380544) [bit](https://twitter.com/parth007_96/status/1604160949434421248) [popular](https://news.ycombinator.com/item?id=34032872).
{% embed url="" %}
This was brought to attention by Charles Frye, fantastic instructor at FSDL
{% embed url="" %}
{% embed url="" %}
{% embed url="" %}
# composer library
\#TIL: Composer looks like an amazing framework with claim to train neural networks up to 7x faster.
# Better to pass callback in fit\_one\_cycle
It’s better to pass callbacks to fine\_tune() and fit\_one\_cycle() method instead of passing directly to the learner object, to avoid unwanted effects when using your models for inferencing. I found this solution from one of the posts written by Wayde Gilliam.
{% embed url="" %}
Zach replied to my tweet, confirming it's correct
{% embed url="" %}
# Eliza - demo
{% embed url="" %}
# Helsinki-NLP Translation models
An open-source repo with lot of pretrained models for translation in more than 1000+ language pairs. If we ever want to build a translation service in our platform, their pretrained models would be a good starting point.
Repository:
Paper: [https://aclanthology.org/2020.eamt-1.61.pdf ]()
Hugging Face:
# Fastai Learning
Having a low training loss but high validation loss does not necessarily mean that your model is already overfitting. This is a point of confusion for most of us when starting out and trying to understand these ideas.
As mentioned in the book; when the validation set accuracy stops getting better, and instead goes in the opposite direction and gets worse, that is when you can start thinking that the model is starting to memorize the training set rather than generalize, and has started to overfit. Taken straight from the book, Chapter 1 :
In other words, as you progress further on the training, once the validation loss starts getting consistently worse than what it was before, (and hence meaning that it’s not able to generalize as well to unseen data, while getting better on training data) then the model has started to overfit.
Not to be taken as an exact example, but I’ve tried to quickly sketch out what this could mean in practice.
and metrics sees no improvement
Source:
# Understanding get\_image\_files in fastai
In **fastai library** we use Datablocks like the below example for loading datasets, and to train models. The below code is DataBlock, which is used to load a dataset of various types of bears to split into train and validation datasets along and to resize images to size 128\*128. For a detailed explanation, check on From Data to DataLoaders section in [Chapter 2 of Fastbook](https://app.gitbook.com/s/-MhiOx8KkYnYxHHqfUbB/python/From%20Data%20to%20DataLoaders).
```
bears = DataBlock(
blocks=(ImageBlock, CategoryBlock),
get_items=get_image_files,
splitter=RandomSplitter(valid_pct=0.2, seed=42),
get_y=parent_label,
item_tfms=Resize(128))
```
In this Datablock `get_items`, we are using the `get_image_files` to load the images. I was curious how to see how `get_image_files` worked under the hood to return all the image files in a dataset. As [Jeremy](https://twitter.com/jeremyphoward) always suggests, I started looking into source code by handy question mark functionality in Jupyter Notebooks. The source code for `get_image_files` can be found in [fastai repo here](https://github.com/fastai/fastai/blob/master/fastai/data/transforms.py). The source code for `get_image_files` function is:
You can see it's expecting the path to the folder where files are present in the image folder. Also the function signature, consists of `recurse=True` and `folder=None` by default.
You can see `get_image_files` function is calling `get_files(path, extensions=image_extensions, recurse=recurse, folders=folders)` on passing with extensions set as `image_extensions`.
> What is image\_extensions doing?
The `image extensions` is just a variable returning a set of images from the [mimetypes](https://docs.python.org/3/library/mimetypes.html), which is part of Python standard library to map filenames to MIME types. Let's see `image_extensions` output to see whole set of image type extensions.
```
>>> image_extensions
{'.jpg', '.svg', '.pgm', '.png', '.xbm', '.jpe', '.pbm', '.pnm', '.rgb', '.tiff', '.xpm', '.jpeg', '.ras', '.ico', '.tif', '.ppm', '.xwd', '.gif', '.bmp', '.ief'}
```
`mimetypes.types_map.items()` returns a dictionary items. It consists of key, value pair and we are selecting value pairs starting with the word `image/` to return a set of image\_extensions as shown in the above output.
Now let's look at `get_files` function, which returns a list of files, based on extensions passed, folders. Let's look into the source code of `get_files(path, extensions=None, recurse=True, folders=None, followlinks=True` function as well:
Let's understand what the function `get_files(path, extensions=None, recurse=True, folders=None, followlinks=True)` is doing line by line.Let's look at the first few lines of code.
```
from fastcore.all import L, setify
from pathlib import Path
def get_files(path, extensions=None, recurse=True, folders=None, followlinks=True):
"Get all the files in `path` with optional `extensions`, optionally with `recurse`, only in `folders`, if specified."
path = Path(path)
folders=L(folders)
extensions = setify(extensions)
extensions = {e.lower() for e in extensions}
```
We are converting the path provided to us into a Pathlib object, and folders are converted to a special Python-like list called (L) based on the `fastcore` library. The extensions are converted to a set if it's being passed as a list, range, string etc. using `setify`. All the extensions are converted to lower case characters if any extension is in upper case. To read more about the `setify` function check [the fastcore docs](https://fastcore.fast.ai/basics.html#setify).
If function definition, we have by default passed `recurse=True`. If it's True, it goes through all the files in the File path we have passed as well as going inside various folders inside the File Path recursively. Else if `recurse=False`, we just go through all files in the File Path we have passed without going inside various folders.
```
import os
def get_files(path, extensions=None, recurse=True, folders=None, followlinks=True):
...
...
if recurse:
...
...
else:
f = [o.name for o in os.scandir(path) if o.is_file()]
res = _get_files(path, f, extensions)
```
For the sake of understanding, let's take an example a `.git` directory, with the following file structure.
`os.scandir` returns an iterator of Directory objects. In Python `os` module, there is an `os.listdir(path='.')` which does the same functionality as `scandir`. Yet `scandir` gives a better performance for most of common use cases. \[1]
```
f = [o.name for o in os.scandir(path) if o.is_file()]
```
It returns a list of file extensions as shown below with list comprehensions, where `is_file()` returns, if it's a file or whether it's pointing to a directory with `followlinks`.
```
>>> path=Path('.git')
>>> [o.name for o in os.scandir(path) if o.is_file()]
['index', 'HEAD', 'packed-refs', 'config', 'description']
```
If `recurse=True`, it goes through all the directories and works on files recursively. Let's look at the sources code and try to understand more.
I would highly recommend the functionality of `os.walk` by checking this [article](https://www.pythonforbeginners.com/code-snippets-source-code/python-os-walk). You can see that on iterating through `os.walk()`, we can get the directory path, and associate file path as a list. This is being passed to `_get_files(p, f, extension)` function.
```
>>> for i,(p,d,f) in enumerate(os.walk(path, followlinks=True)): # returns (dirpath, dirnames, filenames):
... print(p, f)
...
. ['index', 'HEAD', 'packed-refs', 'config', 'description', '.env']
./.gitignore []
./branches []
./hooks ['fsmonitor-watchman.sample', 'update.sample', 'pre-applypatch.sample', 'pre-push.sample', 'pre-receive.sample', 'applypatch-msg.sample', 'pre-commit.sample', 'prepare-commit-msg.sample', 'commit-msg.sample', 'post-update.sample', 'pre-rebase.sample']
./objects []
./objects/pack ['pack-0ae71ead7e875289ae1c9a4b14ca65dbb7a9fc83.pack', 'pack-0ae71ead7e875289ae1c9a4b14ca65dbb7a9fc83.idx']
./objects/info []
./info ['exclude']
./refs []
./refs/tags []
./refs/remotes []
./refs/remotes/origin ['HEAD']
./refs/heads ['master']
./logs ['HEAD']
./logs/refs []
./logs/refs/remotes []
./logs/refs/remotes/origin ['HEAD']
./logs/refs/heads ['master']
```
Just to summarise how the `get_files` function is working it will be useful to look at the below illustration:
When `recurse=False`, for path bears. It returns just returns file README excluding (.gitignore) and directories.
```
>>> get_files(path, recurse=False)
['README']
```
While `recurse=True`, for path bears. It returns all valid files inside the root directory as well as in folders such grizzly, black, teddy, details, folder etc.
After that, it's passed to `_get_files` function, which returns the list of filenames to a list of pathlib Path of various filenames.
`fs`, the list of files returned. We are not passing files that are starting with `.` like `.gitignore or .env` as it's not usually very useful for our dataset to get as files. Also, it's not returning file extensions passed or `f'.{f.split(".")[-1].lower()}'` in extensions.
res on passing `p/f` for the list of files will become a list of paths as shown in the below result. Transforming from a list of file names, we are transforming it to a list of Pathlib module Path, pointing to various filenames.
```
>> get_files(Path('.'))
[Path('index'), Path('HEAD'), Path('packed-refs'), Path('config'), Path('description'), Path('hooks/fsmonitor-watchman.sample'), Path('hooks/update.sample'), Path('hooks/pre-applypatch.sample'), Path('hooks/pre-push.sample'), Path('hooks/pre-receive.sample'), Path('hooks/applypatch-msg.sample'), Path('hooks/pre-commit.sample'), Path('hooks/prepare-commit-msg.sample'), Path('hooks/commit-msg.sample'), Path('hooks/post-update.sample'), Path('hooks/pre-rebase.sample'), Path('objects/pack/pack-0ae71ead7e875289ae1c9a4b14ca65dbb7a9fc83.pack'), Path('objects/pack/pack-0ae71ead7e875289ae1c9a4b14ca65dbb7a9fc83.idx'), Path('info/exclude'), Path('refs/remotes/origin/HEAD'), Path('refs/heads/master'), Path('logs/HEAD'), Path('logs/refs/remotes/origin/HEAD'), Path('logs/refs/heads/master')]
```
This is how the `get_image_files`, returns a L object based on fastcore for any object. For the [BIWI Dataset](https://www.kaggle.com/kmader/biwi-kinect-head-pose-database), the output of `get_image_files` and `get_files` is as following:
## Conclusion
I hope with this blog post, you now have understood how `get_image_files`, fetch the list of images under the hood by looking into the source code.
In case, if I have missed something or to provide feedback, please feel free to reach out to me [@kurianbenoy2](https://twitter.com/kurianbenoy2).
## References
\[1] [Fastai source code](https://github.com/fastai/fastai)
\[2] [Python os module](https://docs.python.org/3/library/os.html#os.listdir)
\[3] [Deep Learning for Coders with Fastai and Pytorch: AI Applications Without a PhD ](https://github.com/fastai/fastbook)
# Resizing an image to smaller size
Resizing images to smaller dimension to satisfy the file size or height/width requirements is something we have faced multiple times. Most of the online tools which I previously used were full of advertisments, and even required to login which is something I hated a lot. Today I learned to do this tedious job with python and it's pretty easy also.
Library used: Pillow
```
import os, sys
from PIL import Image
size = (720, 720)
for infile in sys.argv[1:]:
outfile = os.path.splitext(infile)[0] + f"{size[0]}.jpg"
if infile != outfile:
try:
with Image.open(infile) as im:
im.thumbnail(size)
im.save(outfile, "JPEG")
except OSError:
print("cannot create thumbnail for", infile)
```
Save file as `resize.py` and run the script with:
> python3 resize.py image.jpg
Thanks to [Jishnu](https://twitter.com/jishnu7) and [Rajeesh](https://twitter.com/rajeeshknambiar), I learned there is another tool called ImageMagick. It can also resize images, along with a ton more other cool stuff.
Try image magic, I think it is built into most distros (if not, easy to install via system package manager) and supports more than 200 formats. Easy to use as well.— Jishnu (@jishnu7) [July 20, 2021](https://twitter.com/jishnu7/status/1417363498120605696?ref_src=twsrc%5Etfw)
**To Install ImageMagick in your debian based distro:**
```
sudo apt install imagemagick
```
To resize images with height, width use the option:
```
convert -resize 300x720 image.jpg resized_image.jpg
```
You can resize also based on the quality as follows:
```
convert -quality 50 image.jpg resized_image.jpg
```
# Extracting a Json Object as List(regex)
I had a JSON data like this:
"\[{'3': 'test'}, {'4': 'testathira'}, {'5': 'Sample Test'}, {'6': 'XYZ'}]
I want to write a regex statement to get these object in array of the nested JSON object into python list:
Let's check regex101 with website:
To write pythonic code for the following:
```
import re
x = input()
re.findall(r"{'[0-9]+': '[a-z][A-Z]+'}", x)
```
# f-strings debugging shortcut
Instead of typing things, with it's value twice. With f-strings there is a nice shortcut.
**Without f-strings:**
```
number = 32
print(f"number={number}")
cards = {'R':4, 'J': 24}
print("cards", cards)
```
**With fstrings**
```
print(f"{number = }")
# returns number = 32
print(f"{cards = }")
# returns cards {'R': 4, 'J': 24}
```
# Pytest
Pytest is an awesome framework to test your Python code. It consists of lot of great options to code and make our life way better than it was before
db
session
pytest fixture
# conda switch between python versions
What is `rdkit` and `nb_conda_kernels`? I see you are switching to a virtual environment with different python version. Was it the trick to change kernels you mentioned during walk-thrus session?
**Radek's explaination:**
Yup, that’s it. Essentially, how to have one conda env but multiple python versions. You can have one nb running python 3.5, and another python 3.7. Very rarely is this useful, but in the scenario where you want to run some old python code, there is a solution without having to set up a new environment.
(also, if you have two envs and want to work on the code at the same time or execute files in succession – for instance to process some data in one notebook and continue in the other --, it gets painful… this is just a cleaner way to achieve it, though it is a super rare case that this might be needed)
`rdkit` is just some random repo I wanted to use for a Kaggle competition, not sure I will use it ever again in my life**.**
In fact, this is such a niche case I don’t even think it is worth covering this in the walk-thrus, but probably good to know this solution exists if you ever encounter a situation where it might be helpful (googling for this is not that easy).
Bookmark for conversation:
{% embed url="" %}
# Nested functions exception handling
Original question asked in Real Python slack group:
I have a question about chained exception handling method and how to return except conditions when I use it for one function which relies on another function. The code can be found below for more context:\
\
\
\
It looks to me from answers I got from experienced folks in Real python, there are two ways to do this:
**In main function write try/except code and raise the exception from nested inner function**\
\
Hello this is my rewriten code which required a custom class in Python:\
\
`py`\
`class ModelNotSupported(Exception):`\
`pass`\
\
\
\
\
[**https://gist.github.com/kurianbenoy/346090c325dcf4f8f77ec3fbda3f0f99**](https://gist.github.com/kurianbenoy/346090c325dcf4f8f77ec3fbda3f0f99)
```
supported = ["a", "b", "c", "en", "ta"]
def load_model(source, target):
if source == target:
raise ModelNotSupported("Hello how are you")
if source in supported:
if source == "en":
return ("t", "t")
else:
raise ModelNotSupported("Language pair not available")
def predict_raw():
try:
tokenizer, model = load_model("en", "hello")
return model
except ModelNotSupported as e:
return {"inputerror": e}
```
**This can be be implemented with a function which returns messages, and based on which return based on the conditions.**
```
```
# Installing Linux Operating system
In this documentation, I am writing to brain dump the things each time I do installation.
### My preffered OS
#### I usually prefer to install debian. I prefer to install debian because of it's stability, people I know and the core philosophy of the project. That's why I install it.
I have two free bootware USB sticks. With this it's pretty easy to install anything. The first one was gifted by Abhijit P A, Debian Developer and second one I got it during Debconf.
### Add user to sudoers
Edit: /etc/sudoers\
\
Add username in that file as:\
```
username ALL=(ALL) ALL
```
{% embed url="" %}
### Basic Installation software
```
sudo do apt update
sudo apt upgrade
sudo apt install wget
sudo apt install git
# Installation of chrome
wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
sudo apt install ./google-chrome-stable_current_amd64.deb
```
{% embed url="" %}
### Text Editors
```
sudo apt install vim
```
Cursor.so and VSCODE
{% embed url="" %}
{% embed url="" %}
```
apt install ./<>.deb
#older version
sudo dpkg -i .deb
# sudo apt-get install -f # Install dependencies
```
Configurations:
{% embed url="" %}
{% embed url="" %}
{% embed url="" %}
### Install nodejs
```
sudo apt install nodejs npm
```
{% embed url="" %}
### Install Python
```
wget https://raw.githubusercontent.com/fastai/fastsetup/master/setup-conda.sh
source setup-conda.sh
. ~/.bashrc
conda install -yq mamba -c conda-forge
```
{% embed url="" %}
### Settings
> Check mouse trackpad setting to enable double tap
### Displays in Linux
\
For screensharing - How to use two monitors or share when giving presentations??
It wasn't as complicated as I expected. Just press function key and then press the key for screensharing in laptop.
> In my case it was: FN+ F7 key
# Certbots usage
**certbot certonly --cert-name app2vue.aot-technologies.com**
**Always be a bit hesistant on force renewing a certificate as much as possible, when you notice a database is not disconnected.**
{% embed url="" %}
**Documentation around certbot and it's usage**
Always ensure that the file permissions are set in read+write permission mode.
[**https://www.pluralsight.com/blog/it-ops/linux-file-permissions**](https://www.pluralsight.com/blog/it-ops/linux-file-permissions)
# Code highlighting in Google Docs
A really cool extension for code highlighting in Google Docs! It's called **Code Blocks**, and it's open source and written in JS.
Installation
1. Open addons in google docs and search for codeblock\
\
2\. Install and start using
After adding it to a document, it allows you to format code in Google Docs, with syntax highlighting and a choice of themes! Figured it'd be good help to our teams putting code into google documents!
My colleague Gordon introduced me to this awesome tool.
# HTTP Methods
HTTP defines a set of **request methods** to indicate the desired action to be performed for a given resource.
TRACE - The `TRACE` method performs a message loop-back test along the path to the target resource.\
\
Check:
Other HTTP Methods are:
GET\
POST
PUT
PATCH
PUT is expecting full resource details to modify, while PATCH is expecting with partial details itself to modify stuff
OPTIONS
Using Get request with payload is not a good practise for client applications like in frontend websites, while in RFC protocol nothing is stopping from using that approach.
# How to use vertical mouse?
I bought an mxmaster3 recently, which is one of best mouse out there and one of good quality brand of vertical mouse in India.
How to use?
{% embed url="" %}
Why?
{% embed url="" %}
# HTTP Status Codes
**Flow Diagram of HTTP Decisions**
419 http staus indicates unauthenticated for a valid user whose token might have expired.
Reference:
# Keycloak, Oauth, OpenID connect, SAML
Keycloak is an authentication framework provided Redhat.
**OAuth 2.0:** If you’ve ever signed up to a new application and agreed to let it automatically source new contacts via Facebook or your phone contacts, then you’ve likely used OAuth 2.0. This standard provides secure delegated access. That means an application can take actions or access resources from a server on behalf of the user, without them having to share their credentials. It does this by allowing the [identity provider (IdP)](https://www.okta.com/identity-101/why-your-company-needs-an-identity-provider/) to issue tokens to third-party applications with the user’s approval.
**OpenID Connect:** If you’ve used your Google to sign in to applications like YouTube, or Facebook to log into an online shopping cart, then you’re familiar with this authentication option. OpenID Connect is an open standard that organizations use to authenticate users. IdPs use this so that users can sign in to the IdP, and then access other websites and apps without having to log in or share their sign-in information.
**SAML:** You’ve more likely experienced SAML authentication in action in the work environment. For example, it enables you to log into your corporate intranet or IdP and then access numerous additional services, such as Salesforce, Box, or Workday, without having to re-enter your credentials. SAML is an XML-based standard for exchanging authentication and authorization data between IdPs and service providers to verify the user’s identity and permissions, then grant or deny their access to services.
Group in Keycloak is not part of authentication protocol provided by openid
Scopes - space seperated list of identifiers used to specify what privileges are requestion
Claims - information of user
Response type - Authorization, implicit, explicit
{% embed url="" %}
{% embed url="" %}
# Why should NPM packages be as small as possible?
Most of the popular NPM packages are very small in size, as it's a crucial thing that affects the npm packages. Let's look at why it's a crucial thing:
One is the performance factor of the applications you are building. Most of the applications you are working on are to be highly performant, and should work even in low bandwidth settings.
Imagine if your npm package let's say is 10 MB in size. I don't expect 99% of users in India never load your application in web or mobile build with node\_modules. But Why?
It's because loading your application will easily take 5-10 seconds, while an average application is expected to return a meaningful content load usually in milliseconds. This is why it's very important to keep your application small in size, and easily usable for a lot of folks. This is why popular NPM packages like React, moment.js are just Kbs in size.
One of the most common complaints, I heard from other folks was why to use React for a TodoList App with 3MB size when you can do the same thing easily with vanilla JS + CSS(which is in a few KBs) for a very performant reason. Any frontend framework has its pros and cons.
Also, it's very important to make your packages as small as possible to be highly reliable and cost less cost for computing required for hosting. So one of the simplest things we can do is to reduce dependencies as much as possible.
This can be done by not trying to install third-party dependencies when we are working on any application. Even though I agree a lot of stack-overflow answers recommends this approach. Yet generally the loading time is not so slow even though there are dependencies, as most of the common dependencies like JQuery which are commonly used in multiple websites are cached in the users' browser itself if they come from CDN.
Adam Polak recommends how to reduce [size of node modules for Cost and performance reasons](https://tsh.io/blog/reduce-node-modules-for-better-performance/).
I would like to thank [Faiz Shabeeb](http://shabeebk.com/blog/) for sharing this knowledge.
# Clean Architecture
It contains four layers, yet it's not just limited to it you can extend to as much as possible. The four layers are:
* Entities - like base class models for business code
* Use Cases/ Injectors
* Controllers
* External Interfaces\
\
Then the dependency injection aspect which takes one components, allows that components to be reused anywhere without any hesistancy.\
\
\
### References
1.
# REST vs gRPC
Both REST and gRPC are protocols for communication.
REST is a way client-server communication based on HTTP/1 w/ GET, PUT, POST, DELETE, PATCH methods. Roy Fielding introduced with a set of concepts like cachability, stateless behavior etc.
References
{% embed url="" %}
{% embed url="" %}
gRPC
[gRPC (Remote Procedure Call) is an open-source data exchange technology](https://www.baeldung.com/grpc-introduction) developed by Google using the [HTTP/2 protocol](https://www.baeldung.com/netty-http2).
**It uses the** [**Protocol Buffers binary format (Protobuf) for data exchange**](https://www.baeldung.com/spring-rest-api-with-protocol-buffers#1-introduction-to-protocol-buffers). Also, this architectural style enforces rules that a developer must follow to develop or consume web APIs.
RPC follows a client-response model of communication for [designing web APIs that rely on HTTP/2](https://www.baeldung.com/java-9-http-client). Hence, **gRPC allows streaming communication and serves multiple requests simultaneously**. In addition to that, **gRPC also supports unary communication similar to REST**.
{% embed url="" %}
{% embed url="" %}
# Keycloak concepts
In keycloak, you can
### Mapper
In keycloak, you can create a mapper to add any information from the user attributes, groups, audience etc to be mapped to a corresponding client.The various mappers are shown below with multiple options for customisation to map the items as shown in the below picture like User attributes, Group details, User client role etc.
With a mapper like below redash-group-mapper, we can map any part
Check the video for Understanding keycloak more
{% embed url="" %}
### Client Scopes
It's used to add a reusable scope for adding certain configuration to be allowed to used for certain clients. It can be used along with the mappers we learned to
# what is proxy server and nginx?
Nginx is used in formsflow\.ai to redirect the websites from one ip to another iP in formsflow\.ai. This redirection is being commonly used for formsflow\.ai.
NGINX is a proxy server. What is the function of a proxy server?
1. It acts as an rediction, to protect users password or ip. So instead of google.com, the mapping to be decided for which server to be mapped is being done using proxy.
2. Also it's used for logging user activity and protecting users for certain websites in restrictive companies.
{% embed url="" %}
**NGINX Beginner Guide**
{% embed url="" %}
# Asymptotic Time Complexity
During my datastructures classes in IIIT Kottayam, asymptotic time complexity was explained very well. I thought of writing the example he shared in the class:
`Consider if you are travelling from Kochi to New Delhi. You can reach the location:`
`by flight in 3 hours`
`by train in 2 days`
`by walking in 90 days`
So **big O notation**, can be such that the time required to walk is 90 days. It's considered as the upper bound such that f(n) <= cg(n).
In lowest possible time, let's say we can reach using flight. So it's the lower bound. Such that the f(n)>= c.g(n) always. It's considered **big theta notation**.
Then there is average case complexity which is means between both big O and big theta notation. It gives a tighter bound on run time such that. This is called **big theta notation** and can represented as follows
c1.g(n) <= f(n) <= c2.g(n)
We usually in industry consider even though it's considered as upper bound. It's usually considered closer to definition of big theta time.
# JavaScript Asynchronous operation
* The async/await is a type of promise
* In case of calling a Synchronous function to another Asynchrnous function
* Not really useful
*
* async to async - only useful way of concurrency
*
* Then async to synchrnous - no effect, will be waiting all along
b)
# Lesson 2- Eventloops
* call\_soon
* call\_later
* run\_forever - loops
Select operations
Eventloops can run only one things at a time. There is usually a :
selector or proactor
Reactors - reacts based on notifications used by reactors
twisted calls eventloops
I/0 completion Ports - changes. In windows you can use proactors and selctors. In Unix you can use selectors
Base\_events - consists of base events, yet there is only while loops there
Python 3.5 implementation, then in \`*run*once\`
Heart of heart of asyncio is select\_once method which calls the proactors and is having selector.select(timeout) method
Uvloop for production, as it's doing a lot of magic and improvement in results. As it's fastest implementation with similar performance to NodeJS
Eventloop is just a normal function for handling netoworks requests and is used to handle callbacks. By what?
Eventloop is not magical, it's just a loop which handles network events and executes as callbacks one by one. It can handle many networks events concurrently with a selector or a proactor. While it can handle many callback, but only run one callback at a time. Some callbacks, run and come again with a trick called traumpaline
callsoon, calllater is difficult to run. But it's usually run using coroutine. These coroutines make asynchrnous programs easy and magical
# Lesson 1- asyncio history
Talk about threads in python:
1. Latency - acts and issue. Don't block context in foreground. The code running done in foreground.
Users react using UI is lacky. The car in the background using system.
1.
# Lesson 3- using coroutines
An await - \
will block anything which is to be executed next
asyncio.sleep() - is always atleast how long you are going to wait due to asynchronous nature
difference b/w async function and coroutines
calling await is actually a coroutine
* How the crawling was designed to reduce latency of one thing to another
# Lesson 4- coroutines in hood
Future object can report whether the object is actually being completed or really done
* It's called the promises, deferred in other languages
We looked at about awaitables in asyncio
* coroutines
* tasks
* futures
On looking code implementation of coroutines, it's just simply a generator function. Learned about generators properties like next, fstate etc.
coroutine is just a normal generators, while tasks also doesn't have anything special.
The special functionality is implemented actually really in tasks, where it's calling a loop which calls one callback to another with multiple send loops
In python3, the default implementation is in C, while python implementation is there which is exactly copied in C as well.
**Most common pitfall in asyncio**
Forgetting await with coroutiens. Using type hinting
Holding long task, the blocking should never happen in asyncio
Network lattency in servers. When running 1000 asyncio clients, it's better handling than in other ways.
# Python async/await
#### References
*
*
*
On the plus side, nothing else can run while your code is moving through from one await call to the next, making race conditions harder.
This is called cooperative multitasking, and while it has many upsides, this silent failure mode is its main design issue. If you use a blocking synchronous call by mistake, nothing will explicitly fail, but things will just run mysteriously slowly.
# JavaScript
{% embed url="" %}
# Facet\_grid and Facet\_wrap
ramblings on using ggplot
How to create wonderful visualizations in R like the below with multiple panel plots?
Facetting - can be consider as a method to create multiple panel plots based on splitting attributes which we decide.
**Using facet\_grid**
```
ggplot(gapminder, aes(x=continent, y=gdpPercap))
+ geom_bar(stat = 'identity', fill='forest green')
+ facet_wrap(~year)
```
**Using facet\_wrap**
```
ggplot(gapminder, aes(x=pop)) + geom_density() +
facet_wrap(~year) + scale_x_log10()
```
**Main difference I noticed was:**
\*facet\_wrap\* was working only for a single varaibles, but to create a multple plot in two dimension we had to used \*facet-grid\*
**Reference**
{% embed url="" %}
Dr Ebin Sir's tidverse tutorials
# geom\_point
```
ggplot(gapminder,
aes(x=gdpPercap, y=lifeExp, color=continent)) +
geom_point() + geom_smooth(method='loess', formula='y~x') +
facet_wrap(~continent)
```
For smooth graphs
# Inputting String in C programming language
````
In C programming language, the strings doesn't have a special data type on it's own. So it's usually represented with characters itself.
Usually an array of characters is used to represent characters like this:
``
int char[1000];
``
Characters can be read by variety of methods like:
1. scanf("%s", char);
It can read a single string, but reading strings with spaces is not supported by scanf operators
2. Then there is getline() function which is defined in C programming lanaguage book. It's actually part of
the build in library.
```
#include
#include
int main()
{
int bytes_read;
int nbytes = 100;
char *my_string;
puts ("Please enter a line of text.");
/* These 2 lines are the heart of the program. */
my_string = (char *) malloc (nbytes + 1);
bytes_read = getline (&my_string, &nbytes, stdin);
if (bytes_read == -1)
{
puts ("ERROR!");
}
else
{
puts ("You typed:");
puts (my_string);
printf("%d", strlen(my_string));
}
return 0;
}
```
3. There is gets() functions which is generally considered as unsafe
4. fgets() is more safe as it provides a bounded input
```
fgets(string, 1000, stdin);
```
For more details check [this stackoverflow answer](https://stackoverflow.com/questions/2008173/writing-secure-c-and-secure-c-idioms)
As a bonus, I am sharing how to `find Pangram of a given string in C`
{% highlight linenos %}
#include
#include
#define MAX_LIMIT 1000
int main() {
char string[MAX_LIMIT];
int character_hash[26]={0};
int pancount = 0;
fgets(string, 1000, stdin);
for(int i=0; i>
{% embed url="" %}
# one time setting to easily use Github
I find typing password/remembering tokens hard. So I have been a big fan of setting up ssh keys or the recent trick I found from my colleague Shuhaib.
To clone a repo:
```
git clone https://:@github.com/scuzzlebuzzle/ol3-1.git
```
You can set remote like this way:
```
git remote set-url origin https://:@github.com/scuzzlebuzzle/ol3-1.git
```
**References:**
# Checkout to specific tag
git checkout tags/v1.0 -b v1.0-branch
```
git clone --branch=
```
**Reference:**
1.
2.
# git suggestions in PR
I am always a fan of people who review PR with suggestions like Jesse from Redash team in PRs like the below:
{% embed url="" %}
You can also do the same by following steps mentioned in another TIL:
{% embed url="" %}
# Using emojis in git commits
To use emojis in git commit, you can simply run
git commit -m "My latest fix :sparkles: "
Complete emoji list can be checked out here:
{% embed url="" %}
# Postgres Database Dockercompose
{% embed url="" %}
```
#Author: Kurian Benoy
version: '3.7'
services:
forms-flow-data-analysis-db:
image: "postgres:13"
environment:
POSTGRES_USER: ${DATA_ANALYSIS_DB_USER}
POSTGRES_PASSWORD: ${DATA_ANALYSIS_DB_PASSWORD}
POSTGRES_DB: ${DATA_ANALYSIS_DB_NAME}
POSTGRES_HOST_AUTH_METHOD: "trust"
volumes:
- db-data:/var/lib/postgresql/data
networks:
- forms-flow-data-analysis-api-network
ports:
- '5430:5432'
forms-flow-data-analysis-api:
build:
context: ./
dockerfile: Dockerfile
restart: unless-stopped
links:
- forms-flow-data-analysis-db
ports:
- '5000:5000'
volumes:
- ./:/app:rw
environment:
DATABASE_URL: ${DATA_ANALYSIS_DB_URL}
JWT_OIDC_WELL_KNOWN_CONFIG: ${KEYCLOAK_URL}/auth/realms/${KEYCLOAK_URL_REALM:-forms-flow-ai}/.well-known/openid-configuration
JWT_OIDC_ALGORITHMS: 'RS256'
JWT_OIDC_JWKS_URI: ${KEYCLOAK_URL}/auth/realms/${KEYCLOAK_URL_REALM:-forms-flow-ai}/protocol/openid-connect/certs
JWT_OIDC_ISSUER: ${KEYCLOAK_URL}/auth/realms/${KEYCLOAK_URL_REALM:-forms-flow-ai}
JWT_OIDC_AUDIENCE: ${KEYCLOAK_WEB_CLIENT_ID:-forms-flow-web}
JWT_OIDC_CACHING_ENABLED: 'True'
JWT_OIDC_JWKS_CACHE_TIMEOUT: 300
DATA_ANALYSIS_API_BASE_URL: ${DATA_ANALYSIS_API_BASE_URL}
stdin_open: true # -i
tty: true # -t
networks:
- forms-flow-data-analysis-api-network
networks:
forms-flow-data-analysis-api-network:
driver: 'bridge'
volumes:
db-data:
```
# TIL New SQL Operators - Except, UNION, Distinct
**Except Operator**
The `EXCEPT` operator returns distinct rows from the first (left) query that are not in the output of the second (right) query
**Eg: select \* from table1**
**except**
**select \* from table 2**
Returns all elements in table1 not present in table2
**UNION and Intersect Operator**
In case of two queries in two tables. The Union and Intersect operator does the union and intersection of two sets.
**Distinct Operator**
In case of distinct operator, it returns all the unique values corresponding to a query value we want.
`SELECT DISTINCT(status) FROM employee;`
# Analysing Performance of DB Queries
How can you understand if your Database queries are actually being indexed or not and how efficiently they are searching?
### PostgreSQL
Let's look at how we can analyze the queries in PostgreSQL:
```
explain analyze select id, name from employees where id = 12
```
### MongoDB
In case of mongodb as well there is an explain after collection name which you can use to understand the query performance:
You can use the `cursor.explain()` method or the `db.collection.explain()` method to determine whether or not a query uses an index. On using explain in mongodb queries you can notice that in case of query which requires full search will be in `COLLSCAN status.`
Now in case of efficient searching with indexes, it will be in the stage: IXSCAN
**References**
{% embed url="" %}
{% embed url="" %}
# Querying Date ranges in postgres
When we are trying to query a Date with an PostgresSQL query like:
```
SELECT d, name from table1
where d between '2021-10-01' and '2021-11-31';
```
You will notice your query in Postgres is not including the upper bond ie **2021-11-31.**
**I**norder to resolve this you can query it like below using DATE field with <= and >= operator. This will ensure both the dates are included in the upper bound and lower bound.
```
SELECT d, name from table1
where DATE(d) >= '2021-10-01' and DATE(d) <= '2021-11-31';
```
This query can be implemented in SQLAlchemy as following:
```
db.query(table1.d, table1.name)
.filter(
func.date(table1.d)>='2021-10-01',
func.date(table1.d)<='2021-11-31'
)
```
# Handling Database disconnects in SQLAlchemy
Have you heard about connection pool?
> A connection pool is a standard technique used to maintain long running connections in memory for efficient re-use, as well as to provide management for the total number of connections an application might use simultaneously.Particularly for server-side web applications, a connection pool is the standard way to maintain a “pool” of active database connections in memory which are reused across requests.
Connection pool is a bed rock for handling database connection. And if the database connection is full it can be messy at times.
> A common use case is allow the connection pool to gracefully recover when the database server has been restarted, and all previously established connections are no longer functional. There are two approaches to this.
>
> 1. Disconnet Handling - Pessimistic
> 2. Disconnect Handling - Optimistic
1. **Pessimistic Disconnect Handling**
Let's look at how we can implement pessimistic disconnection in flask-sqlalchemy package. In config, all you need to set is two configurations. In flask setting configuration you will need to set just the following option:
```
SQLALCHEMY_ENGINE_OPTIONS = {
"pool_pre_ping": True,
"pool_recycle": 300,
}
```
> The pessimistic approach refers to emitting a test statement on the SQL connection at the start of each connection pool checkout, to test that the database connection is still viable. Typically, this is a simple statement like “SELECT 1”, but may also make use of some DBAPI-specific method to test the connection for liveness. The approach adds a small bit of overhead to the connection checkout process, however is otherwise the most simple and reliable approach to completely eliminating database errors due to stale pooled connections.\
> \
> The “pre ping” feature will normally emit SQL equivalent to “SELECT 1” each time a connection is checked out from the pool; if an error is raised that is detected as a “disconnect” situation, the connection will be immediately recycled, and all other pooled connections older than the current time are invalidated, so that the next time they are checked out, they will also be recycled before use.
>
> If the database is still not available when “pre ping” runs, then the initial connect will fail and the error for failure to connect will be propagated normally. In the uncommon situation that the database is available for connections, but is not able to respond to a “ping”, the “pre\_ping” will try up to three times before giving up, propagating the database error last received.
**2. Optimistic DB disconnection**
> When pessimistic handling is not employed, as well as when the database is shutdown and/or restarted in the middle of a connection’s period of use within a transaction, the other approach to dealing with stale / closed connections is to let SQLAlchemy handle disconnects as they occur, at which point all connections in the pool are invalidated, meaning they are assumed to be stale and will be refreshed upon next checkout. This behavior assumes the [`Pool`](https://docs.sqlalchemy.org/en/14/core/pooling.html#sqlalchemy.pool.Pool) is used in conjunction with a [`Engine`](https://docs.sqlalchemy.org/en/14/core/connections.html#sqlalchemy.engine.Engine). The [`Engine`](https://docs.sqlalchemy.org/en/14/core/connections.html#sqlalchemy.engine.Engine) has logic which can detect disconnection events and refresh the pool automatically.\
> \
> When the [`Connection`](https://docs.sqlalchemy.org/en/14/core/connections.html#sqlalchemy.engine.Connection) attempts to use a DBAPI connection, and an exception is raised that corresponds to a “disconnect” event, the connection is invalidated. The [`Connection`](https://docs.sqlalchemy.org/en/14/core/connections.html#sqlalchemy.engine.Connection) then calls the [`Pool.recreate()`](https://docs.sqlalchemy.org/en/14/core/pooling.html#sqlalchemy.pool.Pool.recreate) method, effectively invalidating all connections not currently checked out so that they are replaced with new ones upon next checkout. This flow is illustrated by the code example below:
```
from sqlalchemy import create_engine, exc
e = create_engine(...)
c = e.connect()
try:
# suppose the database has been restarted.
c.execute(text("SELECT * FROM table"))
c.close()
except exc.DBAPIError, e:
# an exception is raised, Connection is invalidated.
if e.connection_invalidated:
print("Connection was invalidated!")
# after the invalidate event, a new connection
# starts with a new Pool
c = e.connect()
c.execute(text("SELECT * FROM table"))
```
# WITH NO EXCEPT
Using WITH NO EXCEPT is an intermediate SQL concept to control resource to run(damn I forgot what it was). This was introduced by [Janelle Coburn](https://twitter.com/janelle_coburn) during DataMishaps Nights conference.
{% embed url="" %}
I asked Janelle Coburn via tweet about how to learn some of the intermediate topics in SQL.
{% embed url="" %}
Here reply can be found here:
SQL Shack has some good stuff, I feel like so much of what I've learned is through trial and error (since I'm not a DBA but I've played one on TV sometimes). I would recommend checking out indexes, and query plans.
{% embed url="" %}
One last resource - I have seen Erland speak and I particularly like this article (but he has more)
{% embed url="" %}
As my friend[@PaulAtkinsonPDX](https://twitter.com/PaulAtkinsonPDX)would say, WITH (NOLOCK) isn't a good idea except for when it's a good idea so it needs to be used for a reason. There are some good talks here:
# What is difference with JSON documents in postgres and Mongodb
Mongodb under the hood actually stores the data not in JSON format, but in BSON(which is more efficient in terms of memory and stores the type as well).
While in postgres, what is actually happening under the hood is it's actually a string stored in JSON document. And any SQL engine is actually storing the data and parsing with a JSON parser, but is not internally stored actually as JSON.
So this makes Mongodb approach more faster, scalable when working with JSON or any unrepresented form of data than using postgres databases.