Production Engineering is an organization of engineers who work with our production services throughout their entire life cycle, from design and architecture, through implementation, deployment, and sustaining operation.

よりメタなんだろうか。どうなんだろう。

Senior SRE をみてみる。

https://www.uber.com/global/ja/careers/list/56139/

Production Engineer と同じジャン。。。

Shopify

Production Engineer - Sr Kafka Engineer

Kafka のことばっか注力してる感じだな。

https://www.linkedin.com/jobs/view/1131575400?trk=d_flagship3_salary_explorer&refId=f7bf60f0-b6a3-47c9-a91d-b44c0221c7c8

This position for the Production Engineering team is a hybrid software/system engineer on the Product Sourcing team. Our team covers the disciplines of site reliability engineering, infrastructure engineering, and developer productivity, all to empower merchants and boost their confidence in Shopify’s products.

Site Reliability Engineering だけでなく、なんでもする感じ。

Facebook

www.facebook.com

Production Engineers at Facebook are hybrid software/systems engineers who ensure that Facebook's services run smoothly and have the capacity for future growth. They are embedded in every one of Facebook's product and infrastructure teams, and are core participants in every significant engineering effort underway in the company. Our team is comprised of varying levels of experience and backgrounds, from new grads to industry veterans. Relevant industry experience is important (Site Reliability Engineer (SRE), Systems Engineer, Software Engineer, DevOps Engineer, Network Engineer, Systems Administrator, Linux Administrator, Database Administrator or similar role), but ultimately less so than your demonstrated abilities and attitude. We sail into uncharted waters every day at Facebook in Production Engineering, and we are always learning.

Uber と似たようなことが書いてある。

career のページで SRE を探したけど出てこなかったので、 Production Engineer に統一しているのかもしれない。

おわり

たいした違いはなくて、呼び方の問題ぽい。

2019-11-16

Unicorn の Metrics

Rails で使われているアプリケーションサーバ、Unicorn。

bogomips.org

Master Process から fork して Worker Process が実際の処理を行う。

Unicorn Worker Killer

一定回数以上リクエストがくるか、一定量以上のメモリを使用した場合、Worker Process を再起動する君。

github.com

設定

config/unicorn.rb

worker_processes: Worker Process の数
timeout: timeoutの秒数

prometheus_exporter

Metrics を取得するため、prometheus_exporter を読み込んでいる。

github.com

同僚の takeyama さんの記事

qiita.com

コードをみてみる。

prometheus_exporter/lib/prometheus_exporter/instrumentation/unicorn.rb

github.com

指定された frequency で pid_file から pid を取得して unicorn_collector.collect で metrics を取得して json で返す。

collect_unicorn_stats でとっているのは以下3つ。

metric[:active_workers_total] = stats.active
metric[:request_backlog_total] = stats.queued
metric[:workers_total] = worker_process_count

worker_process_count は直後の private method で定義されている通りで、単純に pgrep で取ってきている。

      result = `pgrep -P #{pid} -f unicorn -a`
      result.lines.count

stats というのがどこからくるかというと listener_address_stats という private method からきていて、tcp であれば Raindrops::Linux.tcp_listener_stats([@listener_address])[@listener_address] を返している。

Raindrops

はてさて Raindrops とは何か。

bogomips.org

raindrops is a real-time stats toolkit to show statistics for Rack HTTP servers. It is designed for preforking servers such as unicorn, but should support any Rack HTTP server on platforms supporting POSIX shared memory. It may also be used as a generic scoreboard for sharing atomic counters across multiple processes.

ほー。

というわけで

Prometheus Exporter で取れる 3つの Metrics は以下

github.com

workers_total: 'Number of unicorn workers.',
active_workers_total: 'Number of active unicorn workers',
request_backlog_total: 'Number of requests waiting to be processed by a unicorn worker.'

active ってのは実際に処理をしているってことなのかな？queued は unicorn に処理されるのを待っているリクエストの数。

request_backlog_total が高い状態が続いた場合、Timeout となってしまいそうだ。

どう解決すればいいのか

Unicorn の Pod の Replicas そのものを増やす？
Unicorn の Worker 数を増やす？
時間がかかっている処理を高速化する？

Horizontal Pod Autoscaler の Custom Metrics でいけないかな？と思ってググったけどたどり着けず。

kubernetes.io

その他の Metrics

内製の gem が送っているやつがあるようだ。

/proc/pid/stats の cpu time をもってきて、毎秒比較し、それが変化していなければ idle と判定しており、そのほかの metrics はそれに基づいて計算している。その結果をDatadog に送るもの。

おわりに

request_backlog_total がしばしば増えていて、それによってエラーも出ているようなのでまずは Metrics の理解が大事だと思ったのでいろいろみてみた。勉強になった。さてどうしよう。

2019-10-25

英会話2019-10-24

会話自信を持って話そう At a Coffee Shop 7 Advanced

わからなかった単語

Keep the change: An instruction to a taxi driver, waiter etc. to keep the change from the amount handed over as a tip
$9.90 - nine dollarS and ninety cents

感想

なんか急にやたら簡単になったな。。。なんなんだ。。。今までめっちゃ難しかったのに。。。

2019-10-23

英会話2019-10-22

会話自信を持って話そう Shopping 7 Advanced

わからなかった単語

costume noun
- a set of clothes in a style typical of a particular country or historical period.
alligator noun
- a large semiaquatic reptile similar to a crocodile but with a broader and shorter head, native to the Americas and China.
- the skin of the alligator or material resembling it.
alligator costume
bin ???
bargain bin: saleのやつが全部ほうりこまれてるやつ
price-conscious 価格に敏感な

発音

clothes

英語の品詞あらわすやつ

Noun 名詞
Pronoun 代名詞
Verb: 動詞
Adjective: 形容詞
Adverb: 副詞
Preposition 前置詞
Conjunction: 接続詞
Interjection: 間投詞

動詞の活用

past tense: 過去形
past participle: 過去分詞

気づき

画像検索するとすぐわかるな。alligator costume。

女の子はpricessになりたくないらしい。そうなの、、、？

Because I don't want to go as a princess.

2019-09-29

英会話2019-09-28

ビジネス会話 Announcing a Pay Cut

Useful Expressions

The company is in the red.
We are forced to impose a salary reduction.
We are left with no other option but to impose a salary reduction.
We're going to have to cut your pay by...
I know this is not the best news.
We didn’t take this decision lightly.
As soon as we break even, we will get your salaries back to where they were.
How big of a reduction are we talking about?
How long do you expect this pay cut to last?

レッスンで使われたセンテンス

We have a definite plan to make that we can recover in three months, so don't worry, your salary will be back to what it was.

レッスンで使われた単語レッスン

drastically-by force, extreme
deficit-shortage
definite-clear

感想

相変わらずセンシティブなテーマが続いているな。。。

2019-09-27

英会話2019-09-26

ビジネス会話 Firing an Employee

Useful Expressions

This is not an easy task for me to do.
There is no easy way to say this.
I'll cut to the chase.
I'm really sorry, but we will have to let you go.
I'm afraid we will no longer need your services.
We have already prepared a letter of recommendation for you.
We will give you a severance package.
We appreciate the contribution you have made to the company.
We appreciate all the hard work you did for us.
We wish you well in your future endeavors.

レッスンで使われたセンテンス

He has to tell James that the company has to let him go.
Considering your performance, though you are doing your job, yet you did not meet the company's expectations.
I called you because I have a bad news. This is not easy for me to say but we have to let you go.

レッスンで使われた単語

endeavors
severance
compensation

感想

I'm afraid we will no longer need your services.

workingとか仕事ぶりって文脈でyour services っていうんやな。

従業員を解雇する日がくるとは今のところ想像できないなあ。

2019-09-26

英会話2019-09-25

ビジネス会話 Resigning From a Job

Useful Expressions

Unfortunately, I have to tell you that I am leaving the company.
I have come to a decision to leave the company.
I'd like to give you my two weeks' notice.
I'd like to hand in my letter of resignation.
I'd like to submit my resignation.
I really enjoyed my time working here.
I am grateful for all your help and support.
I would appreciate/be grateful if you could write a letter of recommendation for me.

レッスンで使われたセンテンス

She told him that she would be resigning.
Are you sure that you wanted to leave the company?

レッスンで使われた単語

obligation: an act or course of action to which a person is morally or legally bound; a duty or commitment.
resignation: an act of resigning from a job or office.
- resign: voluntarily leave a job or office.

その他

読み。 - resigning : RE-SAYN-NING - resignation: RE-SIG-NEY-SHEN

感想

What is your "dream job"? What is the most appealing thing about that job?

ディスカッションでDream Jobの話をした。いい話だ。