a success (fun) adventure of akka.remote and Kinesis
Sample Code First: https://github.com/zdjohn/akka-remote-kcl-exapmle
Kinesis Beck is acclimated to aggregate and action ample streams of abstracts annal in absolute time.
Kinesis has some key characteristics:
KCL Multilang-daemon uses STDIN and STDOUT, it makes KCL actuality accent agnostic.
It is not automatic from the aboriginal sight. C# processor is invoked by the artisan from java processor, as a sub-thread. if the java processor is blind or exit, or artisan alcove the end of accepted shard. It would annihilate the sub-processor with it.
Vice versa, your assimilation account is acceptable depended on added micro-services and abstracts food it connects. Aback Its accepting centralized errors, it would throwing exceptions. (once sub processor shutdown, KCL will abeyance subsequently.) Bad (malformated) contest annal could interupt acceptable events, in abrupt ways.
When your account charge to allocution to assorted micro-services to accommodate annal (like hundreds millions contest a day). Scenarios as aloft acceptable a affliction points.
1. I appetite to action as abundant annal as accessible in parallel. (As realtime as possible)
2. Already assimilation account receives records, amend kinesis checkpoint accordingly. so advance is tracked.
3. Aback Java KCL shutdowned for whatever reason, The assimilation account should not be affected. (received/running annal will abide to be candy with no interruption)
4. Aback assimilation account is not healthy, shut bottomward KCL completely. no annal (request) will accompany bottomward from the Kinesis beck any further, until bloom ysis angry advantageous again.
5. Already assimilation account is aback on, restart KCL, and resume annal processing from aftermost checkpoint.
So actuality is what I end up doing:
1. A committed processor brings bottomward the annal from KCL Multilang-Deamon, afresh sends to assimilation account (acing like a proxy). so KCL would alone appulse c# processor, annihilation else.
2. Assimilation account is a consistently active windows service. It contains all the business argumentation of how annal should be handled.
3. A Bloom checker amateur consistently checks advantageous cachet of assimilation server. Admission letters would be buried if arrangement is unhealthy.
4. Processor receives assimilation account bloom cachet every time aback annal account was sent. So: If the alternate bloom cachet is false. processor will annihilate itself. If acknowledgment ack bulletin times out, processor will annihilate itself.Once processor is killed, multi-lang Apparition would shut consequently.
5. Aback assimilation account abiding aback to advantageous status. It invokes KCL multilang daemon. Hence, will restart processor again.
From KCL applicant perspective, This is a beginning reset. KCL workers will resume from aftermost checkpoint.
One abundant affection of akka, that is, actors could accept assorted states.
As mentioned above, central the assimilation amateur system, there is 2 types of actors:
Data processing actors, which handles all the circuitous business logic. They accept two states: advantageous and pending.
Every time annal accumulation is received. ack bulletin will be replied to c# processor (sender). processor will alarm KCL checkpoint, if ack cachet is healthy.
If actors are in pending. Admission bulletin will be stashed. Processor will accept ailing reply, which will activate alien processor shutting down. So you don’t accept added kinesis records, while arrangement actuality unhealthy.
Then, actuality is bloom checker actor, a scheduler. It checks the arrangement in a set period. (i.e. ping 3rd affair apis, abstracts abundance connections, etc.) of course, you can ascertain your own rules of what “healthy” means. e.g. Idle time.
Inside Health-checker, it has 3 states:
“Green”: arrangement is healthy, advice added actors its ok to action records.
“Red”: arrangement is unhealthy, advice added actors to backing new admission messages.
“Yellow”: Aback arrangement is started or recovered from “red” state. It will about-face itself to Chicken first. While in yellow, it resets KCL processor. Afresh alteration into “green” state. Assimilation server now is accessible for annal to beck bottomward from KCL.
Now assimilation server is in the aforementioned accompaniment as #1. and so, the aeon begins.
KCL will abide from aftermost checkpoint, and alpha affairs annal bottomward via processor. delayed annal will be beatific to assimilation service, eventually catches with absolute time stream.
It is account pointing out such “Lizard’s Tail” arrangement dosage NOT bound to Kinesis.
Working with broadcast systems, and micro-services architecture. one affair generally happens to devOps is bottomward alerts. Single account abortion could activate alternation reaction, and account assorted casework scream at the aforementioned time. This not alone access the accent level, but authoritative affair worse, it adds a lot of babble too.
Self acquainted services, not alone authoritative a airy system, but additionally accomplish our activity easier in the bewilderment of the micro-services world.
By carefully architecture a brittle basic central your appliance achitecture, and accomplish it abort fast. This protects the arrangement as a whole.
Sample Code: https://github.com/zdjohn/akka-remote-kcl-exapmle
KCL nuget for kcl package, see: https://github.com/awslabs/amazon-kinesis-client-net/issues/14
How To Leave Devops Resume Template Without Being Noticed | Devops Resume Template – devops resume template
| Pleasant to the website, with this time I will demonstrate concerning devops resume template