Shoreline.io Intros Open Source Solutions Library to Deliver
July 18, 2022
Shoreline’s open source solutions library is a collection of Op Packs
that make it easier to diagnose and repair the most common
infrastructure incidents in production cloud environments. Launching
with over 35 Op Packs freely available to the community, the solutions
library addresses issues like JVM memory leaks, filling disks, rogue
processes, and stuck Kubernetes pods, among others.
On-call teams understand that self-healing infrastructure drives higher
availability, fewer tickets and better customer satisfaction. Until
today, the path to incident automation was long and hard. With
Shoreline, developers can now create and share open source Op Packs
built in hours, not months. These pre-built automations and diagnostic
notebooks save time and accelerate the path to increased reliability.
Published and provisioned as open source Terraform modules, each Op Pack
contains everything necessary to solve a specific issue, including
pre-defined metrics, alarms, actions, bots, scripts, and tests. With
Shoreline’s Op Pack library, the community identifies what to monitor,
what alarms to set, and what scripts to run to complete the repair. All
Op Packs are completely configurable and allow cloud operations teams to
decide whether to use full automation or an interactive Notebook for
human-in-the-loop repair. Co-developed with Shoreline customers, the Op
Packs available at launch are based on real world on-call experience at
large enterprises, rapidly growing unicorns, and the largest hyperscalar
“We’re all working in the same cloud environments, yet every company has
to figure out on their own how to automate even commonplace issues, like
filling disks or JVM memory leaks,” said Anurag Gupta, co-founder and
CEO of Shoreline. “Companies can no longer afford to write their own
runbooks or custom code automations from scratch. With Shoreline, every
time someone in our community fixes a problem, everyone else benefits.”
“Shoreline makes our cloud infrastructure more reliable,” said Wojciech
Krupa, CTO and co-founder at Knowde. “The platform automatically manages
tickets, fixes issues, and resizes clusters for us, so we scale up and
down at the right time. The Shoreline Op Pack library presents many more
opportunities to make on-call shifts better for our team, and new op
packs every month will only increase the value of this resource. My team
is much happier now that they have Shoreline.”
The following Op Pack solutions are immediately available, and free to
Shoreline customers. The solutions library will continue to grow each
month as new Op Packs are added by the Shoreline community. With each
additional Op Pack in use by a customer, time is freed up for engineers
to focus on innovation, rather than repetitive, mundane tasks that are
better handled through automation.
Op Packs available at launch include:
node retirement - Gracefully terminate
nodes when marked for retirement by the cloud
pod out of memory (OOM) - Generate
diagnostic information and restart pods that ran out
pods stuck in terminating - Identify,
safely drain, and restart stuck pods.
pods restarting too often - Detect pod
restart loops and capture diagnostics to identify
the root cause.
exhaustion - Clear away failed jobs or pods
that are consuming too many IP addresses.
- Stuck Argo
workflows - Argo makes declaratively
managing workflows easy, but it can leave behind
many stale pods after workflow execution that should
Reduce Toil (on both VMs
- Disk resize
/ disk clean - Disk full incidents can lead
to wide-spread outages and data loss that can damage
customer experiences and lose revenue.
issues - Network related issues are often
hard to diagnose, and can lead to a very bad
experience for customers.
JVM issues - Capture diagnostic information
for intermittent issues that are hard to reproduce
- Server drift
- Restore uniformity when configuration files,
databases, and data sources on your VMs and
- Config drift
- Ensure observed state matches desired state on
your system configuration, e.g. Kubernetes yaml,
Cloud config, etc.
exhaustion - Running out of memory rapidly
degrades customer experience and must be pre-empted.
failures in kern.log - Detect when a disk
has errors or has entirely failed by inspecting the
OS’s kern.log. Automatically capture these events
and kick off fixes such as recycling the VM.
failures in kern.log - Detect when a
network interface has errors or has entirely failed
by inspecting the OS’s kern.log. Automatically
capture these events and initiate fixes such as
recycling the VM.
unreachable - Determine when there are no
endpoints behind your Kubernetes service or these
endpoints have become unreachable.
sharding replica management - Determine
when your elastic search clusters have too few
replicas per shard, and automatically kick off
processing at the edge - Analyze log files
on the box to identify issues that cause production
incidents, and eliminate costs of centralized
- Kafka data
Processing Lag - Restart slow/broken
consumers when systems are falling behind in
processing messages through a queue.
- Kafka topic
management - When the length of your Kafka
topic is too long, applications may begin to break.
consuming too many resources - Determine if
the system is using too much memory or CPU at the
CoreDNS service - CoreDNS, the default
Kubernetes DNS service, can degrade in performance
with too many calls causing massive latency.
Optimize Cloud Spend
pod CPU and memory allocations -
Automatically reduce pod CPU and/or memory limits
that are set too high.
- Reclaim idle
hosts - Mark low utilization virtual
machine instances for inactivity, then terminate
unused EBS volumes / snapshots - Eliminate
costs from unused resources.
- Manage data
transfer costs - Detect increased data
transfer volumes, and pinpoint the reasons.
use of on-demand hosts - Determine if
converting on-demand VMs to reserved instances would
create substantial savings.
container check - Flag any container or pod
running in privileged mode.
- Users with
root access check - Flag any VM or
container which has server processes running as a
user with root permissions.
- Open port
check - Ports can easily be opened
unintentionally in a development environment,
especially port 22 for SSH and port 3389 for remote
from unexpected ports - Detect network
connections on ports that are not found on an
- Process list
check - Ensure the correct server processes
are running, since processes sometimes die silently
or old versions are left running.
cryptocurrency mining operations -
Unauthorized cryptocurrency miners must be stopped
from abusing free tiers of cloud service providers.
Avoid Major Outages
rotation - Sooner or later every company
gets bitten by expired certificates and when they
do, it can cause a catastrophic outage.
- DNS lag
- Trigger rolling restarts of the DNS servers when
they are responding slowly and causing widespread
Companies around the
world rely on Shoreline’s incident automation platform
to resolve common incidents in production, broaden the
team that can safely repair incidents, and perform live
site debugging of new incidents. Pairing this Op Pack
solutions content with the Shoreline platform
accelerates time to value and increases ROI for