DZone Spotlight

Once upon a time, getting insights from your data meant running a cron job, dumping a CSV, and tossing it into a dashboard. It was rough, but it worked. Then came the wave — the “Modern Data Stack.” Suddenly, you weren’t doing data unless you had: A cloud warehouse (Snowflake, BigQuery, Redshift)A pipeline tool (Fivetran, Airbyte, Stitch)A transformation layer (dbt, SQLMesh, Dagster)An orchestrator (Airflow, Prefect, Mage)A BI tool (Looker, Metabase, Mode)A reverse ETL layer (Hightouch, Census)Data quality (Monte Carlo, Soda, Metaplane)Metadata (Atlan, Castor, Amundsen)A side of observability, lineage, CI, semantic layers…? Each tool makes sense on paper. But in reality? You spend more time wiring things up than shipping value. You’re maintaining a house of cards where one tool breaking means five others follow. And half your time is just spent figuring out which piece broke. Let’s be honest: You didn’t ask for this stack. You just wanted data your team could trust — fast, clean, and reliable. The Stack Feels Modern, But the Workflow Feels Broken You’ve probably heard this before: “Composable is the future.” Maybe. But if you're a data lead at a startup or a small team inside a big org, composable often means fragile. Every tool you add means: Another integration pointAnother failure modeAnother vendor to payAnother schema to sync And most importantly? Another context switch for your already-stretched data team. Real Talk: Who’s This Stack Actually Helping? Let’s do a quick gut check: Can your team explain how data flows end-to-end?If something breaks, can you trace the issue without hopping between five dashboards?Does adding a new table feel simple — or terrifying? If you answered “no,” “not really,” or “we're working on it,” you're not alone. The Modern Data Stack was built with good intentions. But for many teams, it’s become a distraction from actually delivering value with data. Where the Shine Wears Off At first, the modern data stack feels like a win. You plug in a few tools, spin up some connectors, and schedule a few models. Dashboards start updating automatically. Life’s good. But after a few months in production? That clean architecture diagram starts looking like spaghetti. One Small Change Can Set Off a Chain Reaction Let’s say your RevOps team adds a new field in Salesforce.No one tells you. Fivetran syncs the change without warning.Downstream, your dbt model now breaks. Airflow marks the DAG as failed.Your dashboards show stale or broken data. Stakeholders start pinging. By the time you track down the root cause, you’ve lost half a day — and probably some trust too. What was supposed to be a modular stack ends up being a fragile Rube Goldberg machine. Each piece works in isolation. But together? Every dependency becomes a liability. Too Many Tools, Not Enough Clarity Here’s the pattern we keep seeing — and maybe you’ve felt it too: Airflow runs the show, but no one wants to touch the DAGs.dbt owns transformations, but models are undocumented and nested.BI tools sit at the end of the pipeline, but freshness is inconsistent.Monitoring tools alert you after stakeholders notice something’s wrong. You end up spending your time stitching logs together, rerunning failed jobs, chasing metadata inconsistencies — instead of actually delivering insights. It’s like trying to cook dinner while fixing the stove, cleaning the fridge, and rebuilding the kitchen. Ownership Gets Messy. Fast. When everything is split across tools, so is ownership. Engineers might manage the pipeline orchestration.Analysts build dbt models and reports.Product teams own the source systems.DevOps handles infrastructure. But no one owns the full flow, from source to insight. That’s why when things break, Slack fills with: “Anyone know why the marketing dashboard is blank?” “Is this a dbt thing or an Airflow thing?” “Should we just revert the model?” Without clear accountability, everyone’s responsible — which means no one is. Testing and Monitoring Still Feel Like Afterthoughts The tooling around observability has improved, but let’s be real: most teams still operate in reactive mode. You might have: A few dbt test macros.Monte Carlo or some other data quality alerts.Warehouse usage dashboards. But here’s the kicker — they only catch issues within their silo. None of them answer the full-picture questions: What broke?Why did it break?Who’s affected downstream?What should we do next? So you get alerts. Lots of them. But not insight. And definitely not peace of mind. Operational Overhead Drains the Team The more tools you have, the more glue code you write.The more sync issues you hit.The more time your senior engineers spend firefighting instead of building. Even simple changes — like renaming a column — can require PRs across three repos, rerunning tests, updating dashboards, and notifying every downstream consumer. It’s a system optimized for scale… even if you’re not operating at that scale yet. And ironically? The teams that are operating at scale — the ones who’ve been burned — are the ones now choosing simpler stacks. Reality Check: It’s Not Just a You Problem If this is starting to feel uncomfortably familiar, good news: you're not behind. You're just seeing through the hype. A lot of data teams are now asking: “Do we really need this many tools?”“What’s the minimal setup that actually works?”“How can we focus more on delivery, and less on duct tape?” That’s exactly what we’ll dig into next — the practical patterns, simple tools, and boring-but-beautiful setups that actually hold up in production. No fluff. No pitch. Just what works. What Actually Works in Production By now, it’s clear: the “modern data stack” promised speed, scalability, and modularity — but too often delivered fragility, overload, and confusion. So what does work? Let’s break down the real patterns and principles that help lean data teams build pipelines that are stable, understandable, and resilient — the kind of setups that quietly power good decisions, without daily drama. Principle 1: Simplicity Wins Let’s start with the unpopular truth: fewer tools, more reliability. The best production systems aren’t the most feature-packed — they’re the ones the team fully understands. Every tool you add adds complexity, surface area for bugs, and cognitive load. What works: One warehouseOne transformation layerOne orchestration layer (or none if you can get away with it)A clear BI layer everyone knows how to use It’s not minimalism for the sake of it. It’s about limiting points of failure. You want to know, when something breaks, I know where to look — and you want everyone on the team to feel the same. Principle 2: Favor Proven Tools Over Trendy Ones You don’t need the hottest new data product. You need tools with: A large user baseGood documentationPredictable behavior This usually means sticking with boring but battle-tested things. Think dbt-core over obscure transformation frameworks, Postgres or BigQuery over the newest distributed lakehouse platform. Tools don’t make your pipeline better. Confidence does. Choose the stack your team can debug at 2 am without guessing. Principle 3: Build Data Like Software Code is tested. Deployed. Monitored. Versioned. Your data stack should follow the same discipline: Transformations live in version-controlled code.Changes are peer-reviewed via pull requests.Tests exist for assumptions: null checks, value ranges, schema shape.Deployments are automated and testable.Failures are visible before they reach the BI layer. This doesn’t mean turning your data team into software engineers. It means treating pipelines as production systems, not spreadsheets with a cron job. For example, we can show a minimal dbt model + test combo, to illustrate what “treating data like code” looks like: SQL -- models/orders_summary.sql SELECT customer_id, COUNT(order_id) AS total_orders FROM {{ ref('orders') } GROUP BY customer_id YAML # tests/schema.yml version: 2 models: - name: orders_summary columns: - name: total_orders tests: - not_null - dbt_expectations.expect_column_values_to_be_between: min_value: 0 Why this helps: It shows readers how to start catching upstream issues with basic tests, without needing complex tooling. Principle 4: Automate Where It Hurts Most Don’t automate everything. Automate where mistakes tend to happen. That usually means: Schema change detection and alertsModel test failures on PRsFreshness monitoring for source dataFailure notifications that include useful context (not just “job failed”) Too many teams chase “AI observability” when what they need is: “Let me know before my dashboards show stale or broken data — and tell me exactly where to look.” If you automate anything, start with that. Principle 5: Ownership Must Be Obvious If it’s not clear who owns what, things break silently. Make ownership explicit for: Each data sourceEach critical modelEach dashboard When something breaks, it should take one message to find the right person, not a Slack chain 20 messages deep. In healthy teams, everyone knows the blast radius of their changes, and no one touches production without knowing who depends on it. Principle 6: Good Data Systems Are Understandable by Humans This one’s often overlooked. A “good” pipeline isn’t just reliable — it’s explainable: The logic is documented (in-code, not buried in Confluence).The lineage is clear.The metrics match what the business expects. This is the difference between a pipeline that works and one that’s trusted. You don’t need automated column-level lineage graphs. You need a data model that someone new can grok in a few hours. That’s what makes a system scale. Bottom Line: Build Less. Trust More. Most data teams don’t need more tools. They need: Clearer logicFewer moving partsSafer deploysBetter defaults It’s not exciting. But it works. And once your systems are this solid, that’s when it finally makes sense to layer in smarter stuff, like AI that helps your team move faster without introducing new complexity. We’ll talk about that next. AI Is Helping — But Not the Way You Think Let’s get one thing straight: AI isn’t here to replace your data team. It’s here to stop them from wasting time on things they hate. That’s it. That’s the pitch. If AI can help your team spend less time debugging broken pipelines, rewriting boilerplate SQL, or answering the same questions over and over — that’s a win. Not “data copilot.”Not “autonomous analyst.” Just useful, boring assistance where it counts. What AI Actually Helps With Today Here’s what we’ve seen working in real-world teams: 1. Faster Debugging (With Context) AI can trace why a pipeline failed, not just that it failed. Instead of getting a “dbt run failed” message and spending 20 minutes digging through logs, you get: “Model orders_summary failed because column order_value is missing in source transactions. This was introduced in commit c8721a.” This saves time. It also reduces context switching, which is half the battle in data work. 2. Auto-Documenting Models Without the Pain Let’s be honest. Writing descriptions in schema.yml is nobody’s favorite task. AI can generate clean, accurate descriptions of what a model does based on the SQL logic, column names, and lineage. Not perfect, but 90% good, and that’s enough to stop the rot of undocumented models. This is the kind of stuff that makes onboarding new team members way easier, without you having to schedule a documentation sprint. 3. Explaining SQL to Stakeholders (and Junior Analysts) Ever had someone ping you with: “Can you explain what this query does?” Instead of rewriting it line-by-line, imagine highlighting the query and getting a plain-English summary like: “This query joins orders and customers, filters for orders in the last 30 days, and returns the top 10 customers by revenue.” Simple. Time-saving. Less cognitive load for you. 4. Detecting Anomalies — With Context, Not Just Alerts AI-powered monitors can do more than scream, “something changed.” They can say: “Revenue dropped 22% yesterday — but only in one region, and only for one product line.” You get an alert that actually tells you where to look, not just that something somewhere is off. And this matters, because traditional tools often overwhelm with noise, while AI can prioritize based on past incidents, usage patterns, or what dashboards people are watching closely. 5. Smart Layer Between Tools — Not Another Tool Here’s where AI works best: As a layer that helps glue everything together. It doesn’t replace your stack. It helps you navigate it: Answer “What changed yesterday?”Suggest who to notify when a model changesCatch duplicate logic across modelsRecommend test coverage for critical models The most helpful AI doesn't add complexity — it reduces it. What AI Doesn't Need To Do It doesn’t need to: Replace your BI toolWrite all your SQL from scratchPredict “all possible data issues” magicallyBuild you a data stack from zero Let’s be real. Data is messy. Context matters. AI can help — if you already have a healthy baseline. Trying to use AI on chaos just gives you faster alerts for things you still don’t understand. It’s Not About the Stack. It’s About the People Running It. If there’s one lesson from teams that consistently win with data, it’s this: Their stack is simple. Their team is strong. Their process is clear. Not “strong” as in huge, strong as in focused. They have just enough engineers to keep things humming. A data analyst who gets the business. A PM or founder who actually cares about clean metrics. That’s it. That’s the secret. The Stack Is the Easy Part You can always swap out tools. Airflow for Dagster? Sure.dbt for SQLMesh? Maybe.Fivetran vs Airbyte? Doesn’t matter if no one’s watching sync health. The hard part is alignment. Ownership. Communication. And, more than anything, having the right people in the right roles. Which brings us to something that most blog posts like this don’t mention: Good Hiring Still Fixes More Than Good Tooling If your pipeline is fragile, your dashboards are always out of sync, or you’re buried in tech debt, it’s not always a tooling problem. Sometimes you just need: A real analytics engineer who thinks in DAGsA data scientist who can explain things clearly, not just code themA PM who understands what makes a “good metric” The hard part? Finding those people, especially if you’re hiring in a specific region or for a niche stack. If you’re in that boat, here’s a useful resource we came across while researching for this post: List of data engineering recruitment agencies in the USA — if you’re growing a team there, it’s a helpful breakdown. Conclusion: Your Stack Should Serve You, Not the Other Way Around So as we wrap up this short blog — nothing flashy, just a real take — here’s the part that’s easy to forget: It’s not about the stack. It’s about how it holds up when no one’s looking. A reliable data setup isn’t built by chasing trends. It’s built by: Choosing tools your team actually understandsFixing the small things before they become big onesLetting go of complexity that doesn’t serve you That’s the difference between a stack you survive… and one you trust. The best stack is the one your team understands, owns, and trusts. That’s the goal. Everything else is optional.

To tee off this presentation, consider a TodosList that contains Todo items. You wish to be able to react to the following events. In any Todoitem, when: The title is changedThe completion status is toggledIn the TodosList, when: A new item is addedAn existing item is removed Diving In Here is a basic representation of the respective domain classes: Java @Data @AllArgsConstructor public class Todo { private UUID id; private String title; private Boolean completed; } Java public class TodosList { private final Collection<Todo> todos = new ArrayList<>(); public Todo add(String title){ Todo todo = new Todo(UUID.randomUUID(), title, false); todos.add(todo); return todo; } public void update(String id, String title){ Todo todo = todos.stream().filter(t -> t.getId().toString().equals(id)).findFirst() .orElseThrow(() -> new RuntimeException("no task found with matching id")); todo.setTitle(title); } public void toggle(String id){ Todo todo = todos.stream().filter(t -> t.getId().toString().equals(id)).findFirst() .orElseThrow(() -> new RuntimeException("no task found with matching id")); todo.setCompleted(!todo.getCompleted()); } public void delete(String id){ Todo todo = todos.stream().filter(t -> t.getId().toString().equals(id)).findFirst() .orElseThrow(() -> new RuntimeException("no task found with matching id")); todos.remove(todo); } } Observing changes in the state of an object when the events of interest described above are triggered can be accomplished through different techniques, which I will review here. A basic solution would require explicitly implementing some kind of observation mechanism, like a listener. Java public interface Listener { void onTitleChanged(Todo todo); void onCompletionChanged(Todo todo); void onItemAdded(Todo entity, Collection<Todo> todos); void onItemRemoved(Todo entity, Collection<Todo> todos); } Then, a concrete implementation of the Listener would perform the action necessary when the events of interest are fired. Different implementations of the Listener interface would be required if different behavior is required. Below is one such implementation, which only acknowledges the event happening by printing its details to the console. Java public class BasicListener implements Listener { @Override public void onTitleChanged(Todo todo) { System.out.printf("Task title changed to %s\n", todo.getTitle()); } @Override public void onCompletionChanged(Todo todo) { System.out.printf("Task completion changed to %s\n", todo.getCompleted()); } @Override public void onItemAdded(Todo entity, Collection<Todo> todos) { System.out.printf("Event: add, entity: %s\n", entity); todos.forEach(System.out::println); } @Override public void onItemRemoved(Todo entity, Collection<Todo> todos) { System.out.printf("Event: remove, entity: %s\n", entity); todos.forEach(System.out::println); } } These two classes represent functionality that needs to be woven together in some way to be able to react to state changes. The easiest (and unfortunately, pretty invasive, too) is to add statements in the TodosList object to invoke methods in the BasicListener when the events of interest are happening. The updated TodosList would therefore look something like this. Java public class TodosList { private final Collection<Todo> todos = new ArrayList<>(); private final Listener listener = new BasicListener(); public Todo add(String title){ Todo todo = new Todo(UUID.randomUUID(), title, false); todos.add(todo); listener.onItemAdded(todo, todos); return todo; } public void update(String id, String title){ Todo todo = todos.stream().filter(t -> t.getId().toString().equals(id)).findFirst() .orElseThrow(() -> new RuntimeException("no task found with matching id")); todo.setTitle(title); listener.onTitleChanged(todo); } public void toggle(String id){ Todo todo = todos.stream().filter(t -> t.getId().toString().equals(id)).findFirst() .orElseThrow(() -> new RuntimeException("no task found with matching id")); todo.setCompleted(!todo.getCompleted()); listener.onCompletionChanged(todo); } public void delete(String id){ Todo todo = todos.stream().filter(t -> t.getId().toString().equals(id)).findFirst() .orElseThrow(() -> new RuntimeException("no task found with matching id")); todos.remove(todo); listener.onItemRemoved(todo, todos); } } Java public class Main { public static void main(String[] args) { TodosList list = new TodosList(); Todo t1 = list.add("wake up"); Todo t2 = list.add("make breakfast"); Todo t3 = list.add("watch tv"); list.update(t2.getId().toString(), "work out"); list.toggle(t1.getId().toString()); list.delete(t3.getId().toString()); } } Putting it all together, the main class may look as described above, and it would certainly do a decent job of capturing all the targeted events and executing the prescribed effects. If multiple listener implementations need to be invoked when these state changes happen, then it would require having a collection of such listeners and calling them all sequentially to dispatch the event data. Java public class AnotherListener implements Listener { @Override public void onTitleChanged(Todo todo) { System.out.printf("[**] Task title changed to %s\n", todo.getTitle()); } @Override public void onCompletionChanged(Todo todo) { System.out.printf("[**] Task completion changed to %s\n", todo.getCompleted()); } @Override public void onItemAdded(Todo entity, Collection<Todo> todos) { System.out.printf("[**] Event: add, entity: %s\n", entity); todos.forEach(System.out::println); } @Override public void onItemRemoved(Todo entity, Collection<Todo> todos) { System.out.printf("[**] Event: remove, entity: %s\n", entity); todos.forEach(System.out::println); } } The TodosList would now need to save a reference to the subscribers in a collection, and invoke them all when an event is patched. Java public class TodosList { private final Collection<Todo> todos = new ArrayList<>(); private final Collection<Listener> listeners = new LinkedList<>(); // register listener public void addListener(Listener listener) { this.listeners.add(listener); } // unregister listener public void removeListener(Listener listener) { this.listeners.remove(listener); } public Todo add(String title){ Todo todo = new Todo(UUID.randomUUID(), title, false); todos.add(todo); listeners.forEach(l -> l.onItemAdded(todo, todos)); return todo; } public void update(String id, String title){ Todo todo = todos.stream().filter(t -> t.getId().toString().equals(id)).findFirst() .orElseThrow(() -> new RuntimeException("no task found with matching id")); todo.setTitle(title); listeners.forEach(l -> l.onTitleChanged(todo)); } public void toggle(String id){ Todo todo = todos.stream().filter(t -> t.getId().toString().equals(id)).findFirst() .orElseThrow(() -> new RuntimeException("no task found with matching id")); todo.setCompleted(!todo.getCompleted()); listeners.forEach(l -> l.onCompletionChanged(todo)); } public void delete(String id){ Todo todo = todos.stream().filter(t -> t.getId().toString().equals(id)).findFirst() .orElseThrow(() -> new RuntimeException("no task found with matching id")); todos.remove(todo); listeners.forEach(l -> l.onItemRemoved(todo, todos)); } } Lastly, the main class would be used as the place to register (and perhaps even unregister) listeners. Java public class Main { public static void main(String[] args) { TodosList list = new TodosList(); // register listeners list.addListener(new BasicListener()); list.addListener(new AnotherListener()); // continue in the same way as before Todo t1 = list.add("wake up"); Todo t2 = list.add("make breakfast"); Todo t3 = list.add("watch tv"); list.update(t2.getId().toString(), "work out"); list.toggle(t1.getId().toString()); list.delete(t3.getId().toString()); } } The main problems with this method is that the concerns of the listener (registration, deregistration and dispatching), must be manually woven into observable, which opens up a lot of opportunities for error creep and therefore requires extensive testing. More importantly, the listener implementation is very tightly coupled to the observable, and hence impossible to reuse in any other situation without major modifications. A slightly more idiomatic approach would be to take advantage of Java's built-in Observer and Observable framework to offload much of the observing concerns, listener registration, and listener deregistration to the framework and just focus on the effects, or more explicitly, focus on the corresponding behavior after events are triggered. This method is as intrusive as the basic idea implemented above, and has actually been deprecated since Java 9, and as a result, I would not even encourage anyone to use it. Java @Getter @AllArgsConstructor public class Todo extends Observable { @Setter private UUID id; private String title; private Boolean completed; public void setTitle(String title) { this.title = title; setChanged(); notifyObservers(this); } public void setCompleted(Boolean completed) { this.completed = completed; setChanged(); notifyObservers(this); } } The setters in the Observable need to be instrumented to notify observers of a state change. The existing Listener implementations can be repurposed into observers by implementing Java's own Observer interface. Java public class BasicListener implements Listener, Observer { @Override public void onTitleChanged(Todo todo) { System.out.printf("Task title changed to %s\n", todo.getTitle()); } @Override public void onCompletionChanged(Todo todo) { System.out.printf("Task completion changed to %s\n", todo.getCompleted()); } @Override public void onItemAdded(Todo entity, Collection<Todo> todos) { System.out.printf("Event: add, entity: %s\n", entity); todos.forEach(System.out::println); } @Override public void onItemRemoved(Todo entity, Collection<Todo> todos) { System.out.printf("Event: remove, entity: %s\n", entity); todos.forEach(System.out::println); } @Override public void update(Observable obj, Object arg) { if (obj instanceof Todo todo) { System.out.println("[Observer] received event -> todo: " + todo); } if (obj instanceof TodosList list) { System.out.println("[Observer] received event -> todos: " + list); } } } The other Observer would take similar modifications to the ones made in the first one. Java public class AnotherListener implements Listener, Observer { @Override public void onTitleChanged(Todo todo) { System.out.printf("[**] Task title changed to %s\n", todo.getTitle()); } @Override public void onCompletionChanged(Todo todo) { System.out.printf("[**] Task completion changed to %s\n", todo.getCompleted()); } @Override public void onItemAdded(Todo entity, Collection<Todo> todos) { System.out.printf("[**] Event: add, entity: %s\n", entity); todos.forEach(System.out::println); } @Override public void onItemRemoved(Todo entity, Collection<Todo> todos) { System.out.printf("[**] Event: remove, entity: %s\n", entity); todos.forEach(System.out::println); } @Override public void update(Observable obj, Object arg) { if (obj instanceof Todo todo) { System.out.println("[**Observer**] received event -> todo: " + todo); } if (obj instanceof TodosList list) { System.out.println("[**Observer**] received event -> todos: " + list); } } } The fact that the notifyObserver(obj, arg) in the Observable takes two objects as parameters makes it difficult to be expressive when using this method, because it becomes challenging to detect what attributes changed in the Observable. Java public class TodosList extends Observable { private final Collection<Todo> todos = new ArrayList<>(); public Todo add(String title){ Todo todo = new Todo(UUID.randomUUID(), title, false); todos.add(todo); setChanged(); notifyObservers(todos); return todo; } public void update(String id, String title){ Todo todo = todos.stream().filter(t -> t.getId().toString().equals(id)).findFirst() .orElseThrow(() -> new RuntimeException("no task found with matching id")); todo.setTitle(title); } public void toggle(String id){ Todo todo = todos.stream().filter(t -> t.getId().toString().equals(id)).findFirst() .orElseThrow(() -> new RuntimeException("no task found with matching id")); todo.setCompleted(!todo.getCompleted()); } public void delete(String id){ Todo todo = todos.stream().filter(t -> t.getId().toString().equals(id)).findFirst() .orElseThrow(() -> new RuntimeException("no task found with matching id")); todos.remove(todo); setChanged(); notifyObservers(this); } } The main class changes pretty dramatically since the Observers need to be registered with each Observable party. Java public class Main { public static void main(String[] args) { TodosList list = new TodosList(); BasicListener basic = new BasicListener(); AnotherListener another = new AnotherListener(); // register listeners list.addObserver(basic); list.addObserver(another); Todo t1 = list.add("wake up"); // register listeners list.addObserver(basic); list.addObserver(another); Todo t2 = list.add("make breakfast"); // register listeners list.addObserver(basic); list.addObserver(another); Todo t3 = list.add("watch tv"); // register listeners list.addObserver(basic); list.addObserver(another); // proceed in the usual manner list.update(t2.getId().toString(), "work out"); list.toggle(t1.getId().toString()); list.delete(t3.getId().toString()); } } As mentioned earlier, this approach may have been cutting-edge in its heyday, but in today's technology landscape, those glory days of the past are far behind. It's certainly an improvement over the previous approach in that the observing responsibility is delegated to the underlying framework, but it lacks the critical versatility of reusability, since it's not easy to use without a lot of customization, and hence any solution in which it is a part of is not easily reusable without major refactoring. I have skipped the details of demultiplexing events arriving in the void update(Observable obj, Objects arg) methods of the Observers because it can get very complex, detecting what attributes changed, so that the correct routing may be dispatched to the Listener methods. So what else is out there that will perform the same role as Observer/Observable, but without the difficulty of use associated with the previous two approaches? Enter Signals. This is a concept which I have used extensively in the JavaScript ecosystem, and its non-existence in the Java universe is pretty saddening. This is an attempt to narrow that gap. Shell // https://mvnrepository.com/artifact/com.akilisha.oss/signals # maven <dependency> <groupId>com.akilisha.oss</groupId> <artifactId>signals</artifactId> <version>0.0.1</version> </dependency> # gradle implementation("com.akilisha.oss:signals:0.0.1") Signals uses the concept of targeting attributes that need observing, and the registration of listeners is then implicitly accomplished in the construction phase by simply accessing these observed attributes. Let me illustrate, because that explanation is certainly not exhaustive. The Todo class in this case clearly shows which attributes are candidates for observation. Java @Getter @AllArgsConstructor public class Todo { private final Signal<String> title = Signals.signal(""); private final Signal<Boolean> completed = Signals.signal(false); @Setter private UUID id; public Todo(String title) { this(UUID.randomUUID(), title, false); } public Todo(UUID id, String title, Boolean completed) { this.id = id; this.title.value(title); this.completed.value(completed); } @Override public String toString() { return "Todo{" + "title=" + title.value() + ", completed=" + completed.value() + ", id=" + id + '}'; } } It's always convenient in the majority of cases to work with data carriers implemented as Java's records to complement classes instrumented with Signal attributes. Although not used in this presentation, TodoItem is nonetheless an example of such a data carrier object. Java public record TodoItem (UUID id, String title, Boolean completed){ public TodoItem(String title){ this(UUID.randomUUID(), title, false); } } Now, instead of explicitly implementing Listener interfaces, the effects of changes to the title and completed attributes of a Todo class can be captured during the construction of the Todo objects in a factory method. Each call to the .observe() method will return a Subscription object, which can be stored and then used later on to cancel the captured effect from getting invoked again (similar to unsubscribing a listener). In this presentation, I will not be using the Subscription object here, so that I may focus on effects. Java @Getter @AllArgsConstructor public class Todo { // code omitted from brevity public static Todo from(String title){ Todo todo = new Todo(title); // observe title attribute - multiple Observer effects can be captured here Signals.observe(() -> System.out.printf("Task title changed to %s\n", todo.getTitle().get())); // observe completed attribute - multiple Observer effects can be captured here Signals.observe(() -> System.out.printf("Task completion changed to %s\n", todo.getCompleted().get())); return todo; } } When observing a Collection or a Map attribute, values are wrapped in the SignalCollection and SignalDictionary classes, respectively, because they have unique distinguishing characteristics that need to be handled differently. In this case, the TodosList needs the todos Collection to be Observable too. Java public class TodosList { private final SignalCollection<Todo> todos = Signals.signal(new ArrayList<>()); public Todo add(String title){ // using factory method to create Todo object Todo todo = Todo.from(title); todos.add(todo); return todo; } public void update(String id, String title){ Todo todo = todos.value().stream().filter(t -> t.getId().toString().equals(id)).findFirst() .orElseThrow(() -> new RuntimeException("no task found with matching id")); todo.getTitle().set(title); } public void toggle(String id){ Todo todo = todos.value().stream().filter(t -> t.getId().toString().equals(id)).findFirst() .orElseThrow(() -> new RuntimeException("no task found with matching id")); todo.getCompleted().set(!todo.getCompleted().value()); } public void delete(String id){ Todo todo = todos.value().stream().filter(t -> t.getId().toString().equals(id)).findFirst() .orElseThrow(() -> new RuntimeException("no task found with matching id")); todos.remove(todo); } } The magic sauce is in the choice of methods in the Signal object used to access the underlying Signal values. There are three categories of values that are Observable. Scalar (anything that is neither a Collection nor a Map)Collection (Lists, Sets, etc)Dictionary (Map) Scalar For all Scalar values, the .value() and .value(arg) methods are used to access and set, respectively, the underlying values without triggering effects. The get() and set(arg) methods, however, will register and trigger effects, respectively. An effect is the behavior triggered when an Observable attribute is changed. Collection For Collection values, the .value() method is used to access the underlying Collection value without triggering effects. The get(), forEach(), and iterator() methods will register effects. The set(arg) and value(arg) methods don't exist since they serve no useful purpose in this case. So, to trigger effects, only three methods are currently instrumented for that purpose in the SignalCollection: add, addAll, and remove. Dictionary For Dictionary values, the .value() method is equally used to access the underlying Map value without triggering effects. The get(), forEach(), and iterator() methods will register effects. The set(arg) and value(arg) methods don't exist since they serve no useful purpose in this case. So, to trigger effects, only three methods are currently instrumented for that purpose in the SignalDictionary: put, putAll, and remove. When observing a Collection or a Map, the Signals.observe() method takes different arguments to accommodate the differences in these categories of classes. In this case, the TodosList registers an effect through the forEach() method in the constructor, and the registered handler receives an event name and the affected entity as parameters. The event name represents the name of the method that triggered the effect. Java public class TodosList { private final SignalCollection<Todo> todos = Signals.signal(new ArrayList<>()); public TodosList() { Signals.observe((event, entity) -> { switch (event) { case "add", "remove" -> System.out.printf("Event: %s, entity: %s\n", event, entity); default -> System.out.printf("todos size: %d\n", todos.get().size()); } }); } // code omitted for brevity } In the illustration above, todos.get() will register the effect, and this effect will be triggered by any add() or remove() invocations on the SignalCollection. The main class will now look vastly cleaner than the previous times. Java public class Main { public static void main(String[] args) { TodosList list = new TodosList(); // continue as normal Todo t1 = list.add("wake up"); Todo t2 = list.add("make breakfast"); Todo t3 = list.add("watch tv"); list.update(t2.getId().toString(), "work out"); list.toggle(t1.getId().toString()); list.delete(t3.getId().toString()); } } The output produced will tell the whole story of what happens when the TodosList and TodoItems are updated in the main method above. Shell todos size: 0 Task title changed to wake up Task completion changed to false Event: add, entity: Todo{title=wake up, completed=false, id=4b2e720e-5510-4f35-bd13-4925ff6c6f57} Task title changed to make breakfast Task completion changed to false Event: add, entity: Todo{title=make breakfast, completed=false, id=8be14779-0ec9-44c4-aa94-572d2d21aac0} Task title changed to watch tv Task completion changed to false Event: add, entity: Todo{title=watch tv, completed=false, id=bd665225-8dba-421c-91d6-0b6fb78f5f75} Task title changed to work out Task completion changed to true Event: remove, entity: Todo{title=watch tv, completed=false, id=bd665225-8dba-421c-91d6-0b6fb78f5f75} Source Code The source code for the above example can be viewed in this GitLab repository for convenience.

Like many of you, I have been playing around with Model Context Protocol (MCP). To dive in, I built a sample MCP server implementation for Azure Cosmos DB with Go. It uses the Go SDK, and mcp-go as the MCP Go implementation. This MCP server exposes the following tools for interacting with Azure Cosmos DB: List databases: Retrieve a list of all databases in a Cosmos DB account.List containers: Retrieve a list of all containers in a specific database.Read container metadata: Fetch metadata or configuration details of a specific container.Create container: Create a new container in a specified database with a defined partition key.Add item to container: Add a new item to a specified container in a database.Read item: Read a specific item from a container using its ID and partition key.Execute query: Execute a SQL query on a Cosmos DB container with optional partition key scoping. Here is a demo (recommend watching at 2x speed) using VS Code Insiders in Agent mode: How to Run Shell git clone https://github.com/abhirockzz/mcp_cosmosdb_go cd mcp_cosmosdb_go go build -o mcp_azure_cosmosdb main.go Configure the MCP server: Shell mkdir -p .vscode # Define the content for mcp.json MCP_JSON_CONTENT=$(cat <<EOF { "servers": { "CosmosDB Golang MCP": { "type": "stdio", "command": "$(pwd)/mcp_azure_cosmosdb" } } } EOF ) # Write the content to mcp.json echo "$MCP_JSON_CONTENT" > .vscode/mcp.json Azure Cosmos DB RBAC Permissions and Authentication The user principal you will be using should have permissions (control and data plane) to execute CRUD operations on the database, container, and items.Authentication: Local credentials. Just log in locally using Azure CLI (az login), and the MCP server will use the DefaultAzureCredential implementation automatically.Or, you can set the COSMOSDB_ACCOUNT_KEY environment variable in the MCP server configuration: JSON { "servers": { "CosmosDB Golang MCP": { "type": "stdio", "command": "/Users/demo/mcp_azure_cosmosdb", "env": { "COSMOSDB_ACCOUNT_KEY": "enter the key" } } } } You are good to go! Now, spin up VS Code Insiders in Agent Mode, or any other MCP tool (like Claude Desktop), and try this out! Local Dev/Testing Start with the MCP inspector: npx @modelcontextprotocol/inspector ./mcp_azure_cosmosdb MCP, MCP Everywhere! While this is not an "official" Cosmos DB MCP server, I really wanted to try MCP with Go and also demonstrate the Go SDK for Azure Cosmos DB. Win-win! After all, why should Pythonistas have all the fun? Huge props to the creator of the mcp-go project, Ed Zynda! MCP (at the moment) does not have an "official" Go SDK (yeah, I know, shocking, right!). But there are discussions going on, and I encourage all Gophers to chime in! MCP is touted as the next big thing in the agentic app landscape. Let's see how that goes. In the meantime, keep building stuff and trying out new things. Try out the MCP server and let me know how it goes!

By Abhishek Gupta CORE

When I first started building real-world projects in Python, I was excited just to get things working. I had classes calling other classes, services spun up inside constructors, and everything somehow held together. But deep down, I knew something was off. Why? My code felt clunky. Testing was a nightmare. Changing one part broke three others. Adding a small feature would make me change 10 different files in the package. I couldn't quite explain why - until a seasoned software developer reviewed my code and asked, "Have you heard of dependency injection?" I hadn't. So I went down the rabbit hole. And what I found changed the way I think about structuring code. In this post, I'll walk you through what dependency injection is, how it works in Python, and why you might want to start using a library like dependency-injector. We'll use real-world examples, simple language, and practical code to understand the concept. If you're early in your dev journey or just looking to clean up messy code, this one's for you. So, What the Heck Is Dependency Injection? Let's break it down. Dependency injection (DI) is just a fancy term for "give me what I need, don't make me create it myself." Instead of a class or function creating its own dependencies (like other classes, database, clients, etc), you inject those dependencies from the outside. This makes it easier to swap them out, especially in tests, and promotes loose coupling. A Quick Real-World Analogy Imagine you're running a coffee shop. Every time someone orders coffee, you don't go build a coffee machine from scratch. You already have one, and you pass it to the barista. That's DI. The barista (your class) doesn't create the coffee machine (the dependency to serve coffee); it receives it from outside. Without DI: A Painful Example Python class DataProcessor: def __init__(self): # DataProcessor creates its own database connection self.db = MySQLDatabase(host="localhost", user="root", password="secret") def process_data(self): data = self.db.query("SELECT * from data") # Process the data... return processed_data See the problem? This works, but here's the problem. Our DataProcessor is tightly coupled to a specific MySQL database implementation. Want to switch databases? You'll have to modify this class.Want to test it with a mock database? Good luck with that. With DI: Clean and Flexible Now, here's the same code with dependency injection. Python class DataProcessor: def __init__(self, database): # Database is injected from outside self.db = database def process_data(self): data = self.db.query("SELECT * from data") # Process the data ... return processed_data # Creating and injecting dependencies db = MySQLDatabase(host="localhost", user="root", password="secret") processor = DataProcessor(db) That's it! The processor now takes its database dependency as a parameter instead of creating it internally. This simple change makes a world of difference. Using a DI Library in Python The simplest form of DI in Python is just passing dependencies through constructors (as we saw above). This is called constructor injection and works great for simple cases. But in larger applications, manually writing dependencies gets messy. That's where the dependency-injector library shines. It gives you containers, providers, and a nice way to manage your dependencies. First, let's install the package: Plain Text pip install dependency-injector Now, let's see it in action in the real world. Imagine we're building a data engineering pipeline that: Extracts data from different sources.Transforms it.Loads it into a data warehouse. Here's how we'd structure it with DI: Python from dependency-injector import containers, providers from database import PostgresDatabase, MongoDatabase from services import DataExtractor, DataTransformer, DataLoader class Container(containers.DeclarativeContainer): config = providers.Configuration() # Database dependencies postgres_db = providers.Singleton( PostgresDatabase, host=config.postgres.host, username=config.postgres.username, password=config.postgres.password ) mongo_db = providers.Singleton( MongoDatabase, connection_string=config.mongo.connection_string ) # Service dependencies extractor = providers.Factory( DataExtractor, source_db=mongo_db ) transformer = providers.Factory( DataTransformer ) loader = providers.Factory( DataLoader, target_db=postgres_db ) # Main application etl_pipeline = providers.Factory( ETLpipeline, extractor=extractor, transfomer=transformer, loader=loader ) Let's break down what's happening here: Containers A container is like a registry of your application's dependencies. It knows how to create each component and what dependencies each component needs. Providers Providers tell the container how to create objects. There are different types: Factory: Creates a new instance each time (good for most services)Singleton: Creates only one instance (good for connections, for example)Configuration: Provides configuration valuesAnd many more... (check out the complete list following the link above) Using the Container Python # Load configuration from a file container = Container() container.config.from_yaml('config.yaml') # Create the pipeline with all dependencies resolved automatically pipeline = container.etl_pipeline() pipeline.run() The magic here is that our ETLPipeline class doesn't need to know how to create extractors, transformers, or loaders. It just uses what it's given: Python class ETLPipeline: def __init__(self, extractor, transformer, loader): self.extractor = extractor self.transformer = transformer self.loader = loader def run(self): data = self.extractor.extract() transformed_data = self.transformer.transform(data) self.loader.load(transformed_data) Wrapping Up Dependency injection might sound like something only big enterprise apps need. But if you've ever tried to write tests or refactor your code and wanted to scream, DI can save your sanity. The dependency-injector library in Python gives you an elegant way to structure your code, keep things loosely coupled, and actually enjoy testing. Final Thoughts Next time you're writing a class that needs "stuff" to work, pause and ask: Should I be injecting this instead? It's a simple shift in mindset, but it leads to cleaner code, better tests, and happier developers (you included). If you found this helpful, give it a clap or share it with a teammate who's knee-deep in spaghetti code. And if you want more articles like this — real talk, real examples — follow me here on DZone. Let's write better Python, one clean service at a time.

As AI-powered applications like chatbots and virtual assistants become increasingly integrated into our daily lives, ensuring that they interact with users in a safe, respectful, and responsible manner is more important than ever. Unchecked user input or AI-generated content can lead to the spread of harmful language, including hate speech, sexually explicit content, or content promoting violence or self-harm. This can negatively affect the user experience and may also lead to legal or ethical problems. Microsoft's Azure Content Safety service provides a powerful toolset for automatically classifying and evaluating text (and image) content for various harm categories. By using it, developers can build safer, more reliable AI-driven applications without implementing complex moderation logic from scratch. In this tutorial, we'll walk through how to integrate Azure Content Safety into a Spring Boot application using Spring AI and filter user messages in real time. The app will check incoming messages against Azure’s classification engine, and only forward them to an OpenAI model (via Spring AI) if they meet safety thresholds. Azure Content Safety Deployment Before integrating Azure Content Safety into the application, you need to deploy the service to your Azure subscription. For this tutorial, we’ll use Terraform to automate the setup. The provided Terraform template will create the following: A resource groupAn Azure Content Safety instance (with free F0 tier by default) Prerequisites To run the Terraform deployment, make sure you have the following tools installed: Azure CLITerraform You will also need to authenticate Terraform to Azure. One recommended way is to use a service principal. You can create one with the following command: Shell az ad sp create-for-rbac --name "terraform-sp" --role="Contributor" --scopes="/subscriptions/<your-subscription-id>" --output json > terraform-sp.json Then export the required environment variables: Shell export ARM_CLIENT_ID="appId from terraform-sp.json" export ARM_CLIENT_SECRET="password from terraform-sp.json" export ARM_TENANT_ID="tenant from terraform-sp.json" export ARM_SUBSCRIPTION_ID=$(az account show --query id -o tsv) Deploying With Terraform The deployment is defined in a set of Terraform configuration files. The Terraform template (main.tf) looks like this: Plain Text terraform { required_providers { azurerm = { source = "hashicorp/azurerm" version = ">=4.1.0" } } } provider "azurerm" { features {} } resource "azurerm_resource_group" "rg" { name = var.resource_group_name location = var.location } resource "azurerm_cognitive_account" "content_safety" { name = var.cognitive_account_name location = azurerm_resource_group.rg.location resource_group_name = azurerm_resource_group.rg.name kind = "ContentSafety" sku_name = var.sku_name identity { type = "SystemAssigned" } tags = { environment = "dev" } } output "content_safety_endpoint" { value = azurerm_cognitive_account.content_safety.endpoint } output "content_safety_primary_key" { value = azurerm_cognitive_account.content_safety.primary_access_key sensitive = true } resource "local_file" "content_safety_output" { filename = "${path.module}/outputs.json" content = jsonencode({ endpoint = azurerm_cognitive_account.content_safety.endpoint key = azurerm_cognitive_account.content_safety.primary_access_key }) file_permission = "0600" } The variables.tf file looks like this: Plain Text variable "resource_group_name" { description = "The name of the resource group" type = string default = "rg-content-safety" } variable "location" { description = "Azure region" type = string default = "westeurope" } variable "cognitive_account_name" { description = "Globally unique name for the Cognitive Services account" type = string default = "contentsafetydemo123" # must be globally unique } variable "sku_name" { description = "SKU name for the account (e.g., F0 for Free Tier)" type = string default = "F0" } And here is an example terraform.tfvars for deploying to the free tier: Plain Text resource_group_name = "rg-my-content-safety" location = "westeurope" cognitive_account_name = "contentsafetyexample456" sku_name = "F0" After defining these files, you can initialize and apply the deployment as follows: Shell terraform init terraform apply At the end of the process, Terraform will output the Azure Content Safety endpoint and key. These values will also be saved to a local outputs.json file, which will be used later during the application configuration. Dependencies Since spring-ai is still only available as a milestone release, you need to add the Spring Milestone repository to your Maven configuration. You can do this by including the following snippet in your pom.xml: XML <repositories> <repository> <id>central</id> <name>Central</name> <url>https://repo1.maven.org/maven2/</url> </repository> <repository> <id>spring-milestones</id> <name>Spring Milestones</name> <url>https://repo.spring.io/milestone</url> <snapshots> <enabled>false</enabled> </snapshots> </repository> </repositories> You also need to include the spring-ai-bom in your dependency management section: XML <dependencyManagement> <dependencies> <dependency> <groupId>org.springframework.ai</groupId> <artifactId>spring-ai-bom</artifactId> <version>1.0.0-M6</version> <type>pom</type> <scope>import</scope> </dependency> </dependencies> </dependencyManagement> After that, you can add the following dependencies: spring-boot-starter-web – for building the REST APIspring-ai-openai-spring-boot-starter – for integrating with OpenAI via Spring AIazure-ai-contentsafety – the official Azure Content Safety client library for Java Here's how to include them in your pom.xml: XML <dependencies> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> </dependency> <dependency> <groupId>org.springframework.ai</groupId> <artifactId>spring-ai-openai-spring-boot-starter</artifactId> </dependency> <dependency> <groupId>com.azure</groupId> <artifactId>azure-ai-contentsafety</artifactId> <version>1.0.11</version> </dependency> </dependencies> Setting Up the Chat Client To get an OpenAI ChatClient working, the first step is to provide the required API key. This can be done using the following property: YAML spring: ai: openai: api-key: ${OPENAI_API_KEY} The ChatClient itself, which serves as the interface to communicate with the underlying chat model, can be configured like this: Java @Bean("mainChatClient") ChatClient mainChatClient(ChatClient.Builder clientBuilder, List<Advisor> advisors) { return clientBuilder .defaultAdvisors(advisors) .build(); } Note: By default, the ChatClient will use the gpt-4o-mini model, but this can be overridden using the spring.ai.openai.chat.options.model property. If you wire the ChatClient into a RestController like this, you can start interacting with the chat model through the /chat/send endpoint: Java @RequestMapping("/chat") @RestController public class ChatController { private final ChatClient chatClient; public ChatController(@Qualifier("mainChatClient") ChatClient chatClient) { this.chatClient = chatClient; } public record UserMessage(String message) { } public record AiMessage(String message) { } @PostMapping(value = "/send") public ResponseEntity<AiMessage> sendMessage(@RequestBody UserMessage userMessage) { return ResponseEntity.ok(new AiMessage(chatClient.prompt(userMessage.message()).call().content())); } } As shown in the ChatClient configuration above, you can register so-called advisors to a given client. But what are these advisors? Advisors in Spring AI are components that participate in a chain, allowing you to customize how chat requests are built and how responses are handled. Each advisor in the chain can add logic before the request is sent to the model or after the response is returned. They are useful for injecting system instructions, adding safety checks, or modifying the prompt dynamically without putting this logic directly into the main application code. So we have the ability to create our own Advisor implementation that can check each user prompt using the Azure Content Safety service, based on its harm categories, before the prompt reaches the ChatModel. Harm Categories Azure Content Safety can analyze a user message and classify it into one or more predefined harm categories. These categories represent different types of potentially dangerous or inappropriate content. The classification is multi-label, meaning a single message can be associated with multiple categories at once. There are four built-in harm categories: Hate – Targets or discriminates against individuals or groups based on attributes such as race, ethnicity, nationality, gender identity, sexual orientation, religion, immigration status, ability status, personal appearance, or body size.Sexual – Covers content related to sexual acts, anatomy, romantic or erotic expressions, sexual assault, pornography, or abuse.Violence – Describes or encourages physical harm, injury, or the use of weapons, including violent threats or references to violent acts.Self-harm – Includes language about hurting oneself or suicide. Each harm category comes with a severity level rating. These levels indicate how serious the content is and help you decide what kind of action to take. The severity score ranges from 0 to 7. By default, Azure groups these values into four simplified levels: Scores between 0 and 1 → level 0Scores between 2 and 3 → level 2Scores between 4 and 5 → level 4Scores between 6 and 7 → level 6 However, if you need more fine-grained detail, you can configure the API request to return the original 0–7 scale instead of the simplified version. To configure the Azure Content Safety client and enable harm category–based filtering, I defined the following properties in the application: YAML contentsafety: azure-content-safety: enabled: true endpoint: ${AZURE_CONTENT_SAFETY_ENDPOINT:} key: ${AZURE_CONTENT_SAFETY_KEY:} category-thresholds: -"Hate": 1 -"SelfHarm": 1 -"Sexual": 1 -"Violence": 1 In the above configuration: enabled: Controls whether the Azure Content Safety check is active.endpoint: URL of the deployed Azure Content Safety resource. This value can be found in the outputs.json file generated by the Terraform deployment.key: Access key for authenticating requests. This is also available in the outputs.json file.category-thresholds: Defines the severity level threshold for each harm category. If the severity of any category in a user message exceeds the configured threshold, the message will be rejected. Instead of simply returning a static response like "Rejected message!" when a prompt violates one or more harm category thresholds, the ContentSafetyAdvisor used in this application provides a short explanation. This explanation is generated using a separate ChatClient, which receives the original user message along with the harm category results returned by Azure Content Safety. This secondary client is configured with system instructions that help it generate a polite and informative response. Here's how the rejectExplanationChatClient is set up: Java @Bean("rejectExplanationChatClient") ChatClient rejectExplanationChatClient(ChatClient.Builder clientBuilder) { return clientBuilder .defaultSystem(""" You are a helpful assistant who can give a polite message rejection explanation. All the user messages are checked for content safety by a dedicated service. If the message is rejected, you should explain why the message is rejected. You will get the original message and the results of the content safety service where the content safety result will show the severity of the message in different categories. The severity values are on a scale from 0 to 7, where 0 is the least severe and 7 is the most severe. """) .build(); } The ContentSafetyAdvisor, which is responsible for checking incoming user messages and returning a rejection message if the harm category thresholds are exceeded, looks like this: Java public class ContentSafetyAdvisor implements CallAroundAdvisor { private static final Logger LOGGER = LoggerFactory.getLogger(ContentSafetyAdvisor.class); private static final PromptTemplate REJECTION_PROMPT_TEMPLATE = new PromptTemplate(""" Explain politely why the message is rejected. The rejected message is: {message} The content safety result is: {contentSafetyResult} """); private final ChatClient rejectExplanationChatClient; private final ContentSafetyClient contentSafetyClient; private final Map<String, Integer> categoryThresholds; private final ObjectMapper objectMapper; private final int order; public ContentSafetyAdvisor(ContentSafetyClient contentSafetyClient, ChatClient rejectExplanationChatClient, Map<String, Integer> categoryThresholds, ObjectMapper objectMapper, int order) { this.contentSafetyClient = contentSafetyClient; this.rejectExplanationChatClient = rejectExplanationChatClient; this.categoryThresholds = categoryThresholds; this.objectMapper = objectMapper; this.order = order; } @Override public AdvisedResponse aroundCall(AdvisedRequest advisedRequest, CallAroundAdvisorChain chain) { try { var analyzeTextResult = analyzeText(advisedRequest.userText()); if (!isMessageSafe(analyzeTextResult)) { var rejectExplanation = provideRejectExplanation(advisedRequest.userText(), analyzeTextResult); return createResponse(rejectExplanation, advisedRequest.adviseContext()); } } catch (Exception e) { return createResponse("I'm sorry, I can't answer you now.", advisedRequest.adviseContext()); } return chain.nextAroundCall(advisedRequest); } @Override public String getName() { return "ContentSafetyAdvisor"; } @Override public int getOrder() { return this.order; } private AnalyzeTextResult analyzeText(String text) throws Exception { var request = new AnalyzeTextOptions(text); // request.setOutputType(AnalyzeTextOutputType.EIGHT_SEVERITY_LEVELS); Severity levels from 0 to 7 // request.setOutputType(AnalyzeTextOutputType.FOUR_SEVERITY_LEVELS); Severity levels in {0, 2, 4, 6} (the default). var result = contentSafetyClient.analyzeText(request); LOGGER.info("AnalyzeTextResult of message '{}': {}", text, objectMapper.writeValueAsString(result)); return result; } private boolean isMessageSafe(AnalyzeTextResult analyzeTextResult) { for (var categoryAnalysis : analyzeTextResult.getCategoriesAnalysis()) { if (categoryAnalysis.getSeverity() > categoryThresholds.getOrDefault(categoryAnalysis.getCategory().getValue(), Integer.MAX_VALUE)) { return false; } } return true; } private String provideRejectExplanation(String message, AnalyzeTextResult analyzeTextResult) throws JsonProcessingException { return rejectExplanationChatClient.prompt(REJECTION_PROMPT_TEMPLATE .create(Map.of( "message", message, "contentSafetyResult", objectMapper.writeValueAsString(analyzeTextResult)))) .call().content(); } private AdvisedResponse createResponse(String responseMessage, Map<String, Object> adviseContext) { return new AdvisedResponse(ChatResponse.builder() .generations(List.of(new Generation(new AssistantMessage(responseMessage)))) .build(), adviseContext); } } Message Filtering As shown in the configuration above, I set the default category-threshold for each harm category to a low value of 1. This allows me to demonstrate both an accepted and a rejected message without needing to include anything truly offensive. Below are two simple examples: An accepted message A rejected message Conclusion I hope this short article on integrating Azure Content Safety was helpful for you. The full source code is available here on GitHub. While this tutorial focused on the built-in harm categories provided by Azure, it's also possible to train and use custom categories. But maybe that’s a topic for another time.

I’ve spent years building data pipelines and connecting project management to technical workflows. Disconnected systems lead to manual errors and delays, problems that Jira’s API helps solve. This tool lets code interact directly with project boards, automating tasks such as creating tickets when data checks fail or updating statuses after ETL. For data engineers, the API bridges Jira and databases. Extract issue details into warehouses, build dashboards linking pipeline performance to project effort, or trigger data workflows from Jira events. It’s about seamless integration. In this guide, I want to walk you through how I use the Jira API in my day-to-day data engineering work. We'll cover everything from basic authentication to creating issues automatically, querying data using JQL, and even syncing custom fields with databases. My goal is to give you practical, hands-on examples you can adapt for your own pipelines, helping you connect your project management directly to your database and data processing workflows. Let me show you how to automate tasks, extract the data you need, and integrate Jira smoothly into your data engineering stack. Setting Up Your Jira API Environment First, create an API token in Jira Cloud: Go to your Atlassian account settings.Under Security, generate an API token. Here’s how to authenticate using Python: Python import requests auth = ("your-email@domain.com", "API_TOKEN") For cURL: Python curl -u "email:API_TOKEN" https://your-domain.atlassian.net/rest/api/3/issue I always test connectivity by fetching basic project data: Python response = requests.get("https://your-domain.atlassian.net/rest/api/3/project", auth=auth) print(response.json()) Creating Issues Programmatically Use this JSON template to create tickets. Replace PROJECT_KEY and ISSUE_TYPE_ID with your project’s values (found via Jira’s metadata API): JSON { "fields": { "project": { "key": "PROJECT_KEY" }, "summary": "Data pipeline failure", "issuetype": { "id": "10001" }, "description": { "type": "doc", "content": [{"type": "text", "text": "Alert from our monitoring system"}] } } } Send it via Python: Python url = "https://your-domain.atlassian.net/rest/api/3/issue" headers = { "Content-Type": "application/json" } response = requests.post(url, json=issue_data, headers=headers, auth=auth) Querying Data for Analytics Extract ticket data using JQL (Jira Query Language). This example fetches all bugs from the last 7 days: Python jql = "project = PROJECT_KEY AND issuetype = Bug AND created >= -7d" response = requests.get( f"https://your-domain.atlassian.net/rest/api/3/search?jql={jql}", auth=auth ) Store results in a PostgreSQL database: Python import psycopg2 data = response.json()["issues"] conn = psycopg2.connect("dbname=etl user=postgres") cur = conn.cursor() for issue in data: cur.execute("INSERT INTO jira_issues VALUES (%s, %s)", (issue["key"], issue["fields"]["summary"])) conn.commit() Syncing Custom Fields With Databases Jira’s database schema (source) uses tables like customfield and jiraissue. While direct database access isn’t recommended, here’s how to map API data to SQL: 1. Fetch custom field metadata: Python custom_fields = requests.get("https://your-domain.atlassian.net/rest/api/3/field", auth=auth).json() 2. Create a database table dynamically: Python columns = ["issue_key VARCHAR PRIMARY KEY"] for field in custom_fields: columns.append(f"{field['id']} VARCHAR") cur.execute(f"CREATE TABLE jira_custom_fields ({', '.join(columns)})") Automating Workflows Trigger data pipeline runs when Jira tickets update. Use webhooks: 1. Set up a Flask endpoint: Python from flask import Flask, request app = Flask(__name__) @app.route("/webhook", methods=["POST"]) def handle_webhook(): data = request.json if data["issue"]["fields"]["status"] == "Done": # Start your ETL job here return "OK" 2. Configure the webhook in Jira’s settings. When to Work With Jira Consultants While the API is powerful, complex setups like custom schema migrations or large-scale automation might require expertise. Jira consultants often help teams design these systems, like optimizing how ticket data flows into data lakes or aligning Jira workflows with CI/CD pipelines. Troubleshooting Common Issues API Rate Limits Jira Cloud allows 100 requests/minute. Use exponential backoff: Python import time def make_request(url): for _ in range(5): response = requests.get(url, auth=auth) if response.status_code != 429: return response time.sleep(2 ** _) Data Type Mismatches Jira’s timeworked field stores seconds, while many databases use intervals. Convert during ingestion: Python timeworked_seconds = issue["fields"]["worklog"]["total"] timeworked_interval = f"{timeworked_seconds // 3600}:{(timeworked_seconds % 3600) // 60}:{timeworked_seconds % 60}" What I've shown you here really just gets you started. Think of it like learning the basic chords on a guitar; you can play some simple songs now, but there's a whole world of music still out there. These pieces — authenticating, creating issues, searching data, using webhooks – are the essential building blocks. With these under your belt, you can start building some really useful connections between Jira and your data world. For example, you can absolutely turn Jira into a live, real-time data source for your monitoring and reporting. Imagine dashboards that don't just show database performance, but also display the open engineering tickets related to that specific database. You could pull data on how long issues stay in different statuses and feed that into Grafana or Kibana to visualize bottlenecks in your team's workflow. By regularly fetching data via the API, you get a constantly updated picture, much more alive than static reports. And triggering your data pipelines directly from Jira events? That opens up serious automation possibilities. We touched on using webhooks for this. Think about it: a Jira issue moves to 'Ready for Deployment'. A webhook fires, triggering your CI/CD pipeline to deploy a new data transformation script. Or, a new bug ticket is created with a specific tag like 'Data Quality'. A webhook could automatically trigger a diagnostic script to gather more information about the failure and add it to the ticket description. This links project decisions directly to technical actions, cutting out manual steps and delays. Don't overlook the potential locked away in your past Jira tickets either. There are often years of history sitting there. Using the API, you can extract all that historical data — things like how long tasks actually took versus their estimates, which components had the most bugs, or how quickly critical issues were resolved. This historical data is gold dust for analysis and even machine learning. You could train models to better predict how long future tasks might take, identify patterns that lead to recurring problems, or even forecast potential support load based on recent activity. The API is your key to unlocking that historical treasure chest. But the most important piece of advice I can give is "start small". Seriously. Don't try to build a massive, all-singing, all-dancing integration on day one. You'll likely get overwhelmed. Pick one simple, concrete pain point you have. Maybe it's manually creating tickets for ETL failures. Automate just that one thing first. Get it working reliably. See the time it saves you. Feel that little win. Then, pick the next small thing. Maybe it's pulling a weekly report of completed tasks into a database table. Build that. This step-by-step approach keeps things manageable. You learn as you go, build confidence, and gradually create a more connected system that truly works for you and your team. Expand outwards from those small successes. That’s how you make real progress.

PDF and TIFF: Converting Between Document and Image in Java We rarely encounter just one document format in enterprise applications. The longer a system has been in production, the more likely it is that file interoperability becomes a real concern. That’s especially true for file types that sit at the intersection of document and image processing, like PDF and TIFF. TIFF and PDF are both widely used in healthcare, insurance, and legal services (among other industries) where a premium is placed upon long-term file fidelity and visual accuracy. While PDF has a much wider range of use-cases, TIFF holds ground in archival contexts and systems that prefer image-based representations over embedded formatting. Introduction In this article, we’ll look at what it means to convert between PDF and TIFF formats, in both directions. We’ll first break down how these formats store content, why that matters when we’re making a conversion between each format, and what kinds of considerations Java developers should keep in mind when switching between them. Finally, we’ll explore some open-source and third-party APIs that Java developers can use to streamline these conversions in production code. Understanding the (Significant) Structural Differences Between PDF and TIFF PDF and TIFF are both designed for viewing static content, but they couldn’t be more different in terms of composition. TIFF is raster-based —meaning it stores image data pixel by pixel — while PDF is a container that can mix text, images, and vector graphics all in one file. A PDF can be one or more static, raster-based images, which is one way it overlaps with TIFF in terms of real-world functionality. PDFs are designed to support layers, fonts, compression types, and even embedded scripts. TIFF files, on the other hand, lean heavily into visual fidelity and long-term preservation. Many TIFFs use CCITT or LZW compression and are commonly leaned upon for precision, such as single-bit black-and-white image scanning or medical imaging. When we convert between TIFF and PDF, we’re not just changing file extensions — we’re changing how the file's internal contents are represented. That means we must choose how to properly convert PDFs into valid TIFF images, and how to encapsulate TIFF images into a valid PDF structure that still renders correctly across viewers. Converting From PDF to TIFF: Rendering Pages as Image Data When converting a PDF to TIFF, we’re effectively rendering each page of the PDF as an image, then encoding that image in lossless TIFF format. Under the hood, that involves selecting a rendering resolution (DPI), deciding whether to use grayscale, RGB, or black-and-white output, and finding a way to manage multipage TIFF creation if the original PDF has more than one page. Probably the biggest consideration in this direction of our conversion is resolution. A higher DPI results in better visual fidelity, but also larger file sizes. For documents like scanned contracts or forms, a DPI of 200–300 is typically sufficient. For more detailed visual content, we might need to go higher. Converting From TIFF to PDF: Wrapping Image Data in a Page Structure When converting in the other direction — TIFF to PDF — the process flips: we’re taking big, detailed raster images and wrapping them in a valid PDF page structure. This means we’re making decisions about page sizes, margins, orientation, and lossy or lossless compression algorithms (e.g., using JPEG or Flate inside the PDF). If the TIFF we’re converting from is a multipage image, we’ll need to create a separate PDF page for each frame of the TIFF. It's important to understand that we’re not embedding a TIFF inside a PDF — we’re re-encoding its contents into a different format entirely. Open-Source Libraries for PDF-TIFF Conversion For those looking to handle conversions to and from PDF and TIFF with open-source solutions, there are several established libraries that stand out. PDF to TIFF With Apache PDFBox and JAI Apache PDFBox is a robust and well-maintained library for working with PDF files in Java. Most importantly, it includes built-in support for rendering PDF pages to BufferedImage instances, which can then be encoded into TIFF format using the Java Advanced Imaging (JAI) API or standard ImageIO plugins. PDFBox gives us control over rendering DPI, color models, and page iteration, which makes it a solid choice for export workflows. TIFF to PDF With iText or OpenPDF To go from TIFF to PDF, we'll need a library that can build PDFs from image data. To that end, iText (and its fork, OpenPDF) offers a clean way to insert image content into new PDF documents, one image per page. With a multipage TIFF, we'd extract each frame as a separate image and append them sequentially to our resulting PDF. These libraries handle PDF layout and compression settings, which takes a lift off our hands. We can fine-tune the output as needed. These tools don’t abstract away all the complexity, but they’re reliable, well-documented, and used in many production environments. They'll take us where we need to go. Converting PDF to TIFF, TIFF to PDF With Web APIs If we’re not looking to get intimately involved in open-source documentation, we can alternatively try a pair of free plug-and-play conversion APIs that integrate with our Java application using minimal code. Code examples are provided below in this case. These aren’t open-source, which may be a no-go for some, but the idea here is to abstract and simplify all the cumbersome aspects of a high-fidelity conversion away from our environment entirely, including the overhead processing that would otherwise fall on our resources to support. While not particularly relevant here, it's also worth noting that each of these APIs also supports conversions from a variety of other image formats (like WEBP, JPG, PNG, etc.). PDF to TIFF The first thing we'll do is install the SDK with Maven. We'll first add a reference to the repository in pom.xml: XML <repositories> <repository> <id>jitpack.io</id> <url>https://jitpack.io</url> </repository> </repositories> And then we'll add a reference to the dependency in pom.xml: XML <dependencies> <dependency> <groupId>com.github.Cloudmersive</groupId> <artifactId>Cloudmersive.APIClient.Java</artifactId> <version>v4.25</version> </dependency> </dependencies> Following that, we'll add the import classes to the top of our file: Java // Import classes: //import com.cloudmersive.client.invoker.ApiClient; //import com.cloudmersive.client.invoker.ApiException; //import com.cloudmersive.client.invoker.Configuration; //import com.cloudmersive.client.invoker.auth.*; //import com.cloudmersive.client.ConvertApi; Next, we'll initialize the API client, set an API key for authorization (we can get one of these for free), create a conversion API instance, and call the API with our PDF file input: Java ApiClient defaultClient = Configuration.getDefaultApiClient(); // Configure API key authorization: Apikey ApiKeyAuth Apikey = (ApiKeyAuth) defaultClient.getAuthentication("Apikey"); Apikey.setApiKey("YOUR API KEY"); // Uncomment the following line to set a prefix for the API key, e.g. "Token" (defaults to null) //Apikey.setApiKeyPrefix("Token"); ConvertApi apiInstance = new ConvertApi(); File imageFile = new File("/path/to/inputfile"); // File | Image file to perform the operation on. Common file formats such as PNG, JPEG are supported. try { byte[] result = apiInstance.convertToTiff(imageFile); System.out.println(result); } catch (ApiException e) { System.err.println("Exception when calling ConvertApi#convertToTiff"); e.printStackTrace(); } We'll get our TIFF image back as a byte array, which we can then write to a new TIFF file. TIFF to PDF We'll follow similar instructions here. We'll once again install the SDK with our pom.xml reference: XML <repositories> <repository> <id>jitpack.io</id> <url>https://jitpack.io</url> </repository> </repositories> And our pom.xml dependency: XML <dependencies> <dependency> <groupId>com.github.Cloudmersive</groupId> <artifactId>Cloudmersive.APIClient.Java</artifactId> <version>v4.25</version> </dependency> </dependencies> Then we'll add our imports: Java // Import classes: //import com.cloudmersive.client.invoker.ApiClient; //import com.cloudmersive.client.invoker.ApiException; //import com.cloudmersive.client.invoker.Configuration; //import com.cloudmersive.client.invoker.auth.*; //import com.cloudmersive.client.ConvertDocumentApi; Finally, we'll initialize conversion iteration following the same overarching structure as the prior: Java ApiClient defaultClient = Configuration.getDefaultApiClient(); // Configure API key authorization: Apikey ApiKeyAuth Apikey = (ApiKeyAuth) defaultClient.getAuthentication("Apikey"); Apikey.setApiKey("YOUR API KEY"); // Uncomment the following line to set a prefix for the API key, e.g. "Token" (defaults to null) //Apikey.setApiKeyPrefix("Token"); ConvertDocumentApi apiInstance = new ConvertDocumentApi(); File inputFile = new File("/path/to/inputfile"); // File | Input file to perform the operation on. try { byte[] result = apiInstance.convertDocumentAutodetectToPdf(inputFile); System.out.println(result); } catch (ApiException e) { System.err.println("Exception when calling ConvertDocumentApi#convertDocumentAutodetectToPdf"); e.printStackTrace(); } Similar to our other conversion, this will return a PDF byte array, which we can write to a PDF with the proper extension. Conclusion In this article, we explored the relationship between PDF and TIFF formats and how files can be programmatically converted between them. We suggested open-source and non-open-source API solutions to simplify the implementation of this conversion workflow in a Java environment.

By Brian O'Neill CORE

In this post, I'd like to talk a little about scalability from a system design perspective. In the following paragraphs, I'll cover multiple concepts related to scalability—from defining what it is, to the tools and approaches that help make the system more scalable, and finally, to the signs that show whether a system is scaling well or not. What Is Scalability? First things first: I’m pretty sure you know what scalability is, but let’s have a brief look at the definition just to be safe. Scalability is the ability—surprise—of a system or application to scale. It is probably one of the most crucial non-business features of every modern-day piece of code. After all, we all want our systems to be able to handle increasing traffic and not just crash the first chance they can. Besides just making our system work properly under increased load, good scalability offers other benefits: Stable User Experience - We can ensure the same level of experience for an increased number of users trying to use our system.Future Proof - The more we think about the scalability and extensibility of our system during the design phase, the smaller the chance that we will need an architecture rework in the near future.Competitive Advantage - Stable user experience under an extensive increase in traffic can shortly evolve into a business advantage, especially when some competitors’ sites are down while our site is up and running. In the most simple way, when we’re looking at scaling, there are two ways to handle that task—Horizontal and Vertical. Horizontal scaling is focused on adding more nodes to the system, while Vertical scaling is about adding more resources to a single machine that is responsible for hosting our application. However, this is only the beginning, or the end, depending on the perspective. Adding more resources to the physical machines has its physical limits. On the other hand, spawning multiple instances also has its limitations, not to mention significant complexity penalties with the possible architecture redesigns. That is why today we will dive deeper into tools and concepts that will help us make our system more scalable. Before that, let’s take a look at how we can measure the scalability of our system. How To Measure Scalability There are a couple of methods to do that. We can go from an experimental method and set up some stress tests, for example using Gatling, that will show us pretty visible results on how far our systems can scale. On the other hand, you can go for a more quantitative approach and try to calculate the limits of your system. You can read more about the quantitative approach here. As for the experimental methods, I recommend you taking a look at the Gatling Academy. It looks like a nice introduction into writing performance tests. Signs Your System Is Not Scaling Well, there are a couple of metrics and behaviors we can notice, which indicate possible scalability problems within our system: Steadily increasing response times - The response time starts to increase steadily as new users join, after a certain threshold.Higher error rate - The number of erroneous responses, timeouts, dropped messages or connections starts to increase.High cost to low performance - Gaining actual performance benefits (for example by adding more instances) gives little to no results despite investing a relatively high amount of money.Backlog growth - Processing queues, task managers, thread pools, schedulers start failing to keep up with incoming load, extending processing times, possibly later ending up as timeouts.Direct feedback - An important client is calling CTO/CIO or whoever else and complaining that everything is down, alerts start spinning, everything explodes, and other funny events. The Scalability Game Changer There is one key concept that can impact the scalability of our system quite significantly. The concept in question is statelessness, or stateless services/processing. The more stateless services or steps in our workflow we have, the easier it is to scale up the system. We can basically keep spawning new and new instances infinitely—there are some theoretical and of course practical limits, but they are highly dependent on your architecture. You can read more about them here. Of course, this approach has some potential drawbacks. For example, we have to store the state somewhere, since most of the systems are, by definition, stateful. We can offload it to some external dependencies like caches or databases, but this also has its drawbacks. While usually quite scalable, such tools can impose some further penalties on our systems and, what is more, impose some architectural requirements on our systems. That is why we should do in-depth research before actually picking the third-party solutions we want to offload our state to. Nevertheless, in my opinion, the potential benefits of having stateless services overcome their drawbacks, or at least make it an approach worth considering. With this somewhat long dive into the basics of scalability done, we can move to the main course of today's article: The tools and concepts that will make our application scale. Tools For Scalability We can skip captain obvious and move right away past the things like adding more resources to the server or spawning more instances of our service. Caching Creating a cache layer in front of our application is probably the simplest method to increase the scalability of our application. We can relatively easily offload part of the incoming traffic to a cache service that can store frequently queried data. Then, our service can focus either on handling incoming write requests while only handling read requests in case of cache misses and invalidations. Unfortunately, caching also has a couple of drawbacks and considerations that we have to think of before going head-on into adding it to our service. Firstly, cache invalidation is probably one of the biggest problems in the software world. Picking and tuning it correctly may require substantial time and effort. Next, we can take a look at cache inconsistencies and all the possible problems related to handling them correctly. There are a couple more problems related to caching, but these should be enough for a start. I promise to take a more detailed look at caching and all its quirks in a separate blog. Database Optimizations It is not a tool per se. However, it is probably one of the most important things we can do to make our system more scalable. What I exactly mean by this is: Well, sometimes the database can be the limiting factor. It is not a common case; nevertheless, after reaching a certain processing threshold, the chances for it to happen increase exponentially. On the other hand, if you do a lot of database queries (like 10–100 thousands) or your queries are suboptimal, then the chances also increase quite significantly. In such cases, performance tuning of your database can be a good idea. In every database engine there are switches and knobs that we can move up or down to change its behavior. There are also concepts like sharding and partitioning that may come in handy with processing more work. What is more, there is a concept called replication that is particularly useful for read-heavy workflows based on plain SQL databases. We can spawn multiple replicas of our database and forward all the incoming read traffic to them, while our primary node will take care of writes. As a last resort, you can try to exchange your database for some other solution. Just please do proper research beforehand, as this change is neither easy nor simple, and may result in degradation, not the up-scaling, of your database performance. Not to mention all the time and money spent along the way. Usually, even plain old databases like Postgres and MySQL (Uber) can handle quite a load, and with some tuning and good design, should be able to meet your requirements. Content Delivery Network (CDN) Setting up a CDN is somewhat similar to setting up a cache layer despite having quite a different purpose and setup process. While the caching layer can be used to store more or less anything we want, in the case of a CDN it is not that simple. They are designed to store and then serve mostly static content, like images, video, audio, and CSS files. Additionally, we can store different types of web pages. Usually, the CDN is a paid service managed and run by third-party companies like Cloudflare or Akamai. Nevertheless, if you are big enough, you can set up your own CDN, but the threshold is pretty high for this to make sense. For example, Netflix has its own CDN, while Etsy uses Akamai. The CDN consists of multiple servers spread across the globe. They utilize location-based routing to direct the end users to the nearest possible edge node that contains the requested data. Such an approach greatly reduces wait time for users and moves a lot of the traffic off the shoulders of our services. Load Balancing Load balancing is more of a tool for helping with availability than with the scalability of the system. Nonetheless, it can also help with the scaling part. As the name suggests, it is about balancing the incoming load equally across available services. There are different types of load balancers (4th and 7th level) that can support various load balancing algorithms. The point for load balancers and scalability is that we can route requests based on request processing time, routing new requests to less busy services. We can also use location-based routing similar to the case of a CDN. What is more, Load Balancers can cache some responses, integrate with CDNs, and manage the on-demand scaling of our system. On the other hand, using Load Balancers requires at least some degree of statelessness in the connected service. Otherwise, we have to use sticky sessions, which is not desired behavior nor recommended practice. Another potential drawback of using load balancers is that they may introduce a single point of failure into our systems. We all know what the result of such an action may be. The desired approach would be to use some kind of distributed load balancers, where some nodes may actually fail without compromising the whole application. Yet doing so adds additional complexity to our design. As you see, it is all about the trade-offs. Async Communication Here, the matter is pretty straightforward: Switching to async communication frees up processing power in our services. Services do not have to wait for responses and may switch to other tasks while the targets process the requests we already sent. While the exact numbers or performance increase may vary significantly from use case to use case, the principle remains. Besides the additional cognitive load put upon the maintainers and some debugging complexity, there are no significant drawbacks related to switching to an async communication model. This will get more interesting in the next topic. Just to clarify, in this point I thought mostly of approaches like async HTTP calls, WebSockets, async gRPC, and SSE. We will move into the messaging space in the next paragraph. Messaging It is a type of asynchronous communication that uses message queues, message brokers to exchange data between services. The most important change that it brings is the decoupling of sender and consumer. We can just put the message on the queue and everyone interested can read it and act as they want. Message brokers are usually highly scalable and resilient. They will probably handle your workload and not even break a sweat while doing so. Unfortunately, they introduce a few drawbacks: Additional complexity - Messaging is totally different on a conceptual level than the plain request-response model, even if done in an async manner. There may be problems with logging, tracing, debugging, message ordering, messages being lost, and possible duplication.Infrastructure overhead - Setting up and running a platform for exchanging messages can be a challenging undertaking, especially for newcomers.Messages backlog - If the producers send messages faster than the consumer is able to process them, then the backlog grows, and if we do not handle this matter, it may lead to system-wide failures. There is a concept of Backpressure that aims to address this issue. Service Oriented Architecture (SOA) Last but not least, I wanted to spend some time describing any type of service-oriented architecture and its potential impact on how our application scales. While this type of architecture is inherently complex and hard to implement correctly, it can bring our system’s scalability to a totally different level: Decoupling the components enables scaling components in isolation. If one service becomes a bottleneck, we can focus optimization efforts solely on that service.We can find the root cause of potential performance bottlenecks in a more efficient and timely manner, as we can gather more insights into the behavior of different services.Our autoscaling becomes more granular, allowing us to apply tailored scaling strategies on a per-service basis. All of these features improve our efficiency, which can result in reduced costs. We should be able to act faster and more adequately. We also avoid potential over-provisioning of resources across the entire application. However, the sheer amount of potential problems related to this architecture can easily make it a no-go. Why We Fail To Scale After what, how, and why, it is time for why we fail. In my opinion and experience, there are a few factors that lead to our ultimate failure in making an application scalable. Poor understanding of tools we use - We can have state-of-the-art tools; but we have to use them correctly. Otherwise, we should better drop using them—at least we will not increase complexity.Poor choice of tools - While the previous point addresses the improper use of state-of-the-art tools, this one addresses the issue of choosing incorrect tools and approaches for a job, either on a conceptual or implementation level. I kind of tackled the problem with the quantity of different tools for a single task here.Ignoring the trade-offs - Every decision we make has short- and long-lasting consequences we have to be aware of. Of course, we can ignore them; still, we have to know them first and be conscious as to why we are ignoring some potential drawbacks.Bad architecture solutions - Kind of similar point as the one above. This is a point that addresses what happens if we ignore one too many trade-offs—the answer is that our architecture will be flawed in a fundamental way, and we will have to change it sooner or later.Under-provisioning / Over-provisioning - It is the result of loopholes in our data while we are researching what we would need to handle incoming traffic. While over-provisioning is not a scalability failure per se, as we were able to meet the demand, someone may ask, but at what cost? Summary We walked through a number of concepts and approaches to make our application more scalable. Here they are: ConceptDescriptionProsConsCachingIntroduces a cache layer to store frequently accessed data and offload traffic from the application.- Reduces load on the backend - Improves response time - Handles read-heavy workflows efficiently- Cache invalidation is complex - Risk of inconsistenciesDatabase OptimizationsImproves database performance through tuning, sharding, replication, and partitioning.- Handles high read/write workloads - Scalability with replication for read-heavy workflows - Efficient query optimization- Complex setup (e.g., sharding) - High migration costs if switching databases - Risk of degraded performanceCDNDistributes static content (e.g., images, CSS, videos) across globally distributed servers.- Reduces latency - Offloads traffic from core services- Primarily for static content - Can be quite costlyLoad BalancingDistributes incoming traffic across multiple instances.- Balances traffic efficiently - Supports failover and redundancy - Works well with auto-scaling- Requires stateless services - SPOF (if centralized)Async CommunicationFrees up resources by processing requests asynchronously, allowing services to perform other tasks in parallel.- Increases throughput - Reduces idle waiting time- Adds cognitive load for maintainers - Debugging and tracing can be complexMessagingUses message queues to decouple services.- Decouples sender/consumer - Highly scalable and resilient - Handles distributed workloads effectively- Infrastructure complexity - Risks like message backlog, duplication, or loss - Conceptually harder to implementService-Oriented Architecture (SOA)Breaks down the application into independent services that can scale and be optimized individually.- Granular auto-scaling - Simplifies root-cause analysis - Enables tailored optimizations - Reduces resource wastage- High complexity - Inter-service communication adds overhead - Requires robust observability and monitoring Also, here is the table of concrete tools we can use to implement some of the concepts from above. The missing ones are either too abstract or require multiple tools and techniques to implement. ConceptToolCachingRedis, MemcachedCDNCloudFlare, AkamaiLoad BalancingHAProxy, NGINXAsync CommunicationAsync HTTP, WebSockets, async gRPC, SSEMessagingRabbitMQ, Kafka I think that both these tables are quite a nice summary of this article. I wish you luck on your struggle with scalability, and thank you for your time.

By Bartłomiej Żyliński CORE

Index maintenance is a critical component of database administration as it helps ensure the ongoing efficiency and performance of a Structured Query Language (SQL) Server environment. Over time, as data is added, updated, and deleted, index fragmentation can occur, where the logical and physical ordering of index pages becomes misaligned. This fragmentation can lead to increased disk I/O, decreased query performance, and overall system inefficiency. Running index maintenance jobs, such as those provided by the Ola Hallengren SQL Server Maintenance Solution, allows DBAs to proactively address this fragmentation and optimize the indexes for better performance. By regularly monitoring index fragmentation levels and executing maintenance operations like index reorganizations and rebuilds, DBAs can keep their databases running at peak efficiency. This is especially important for large, mission-critical databases, where any degradation in performance can have a significant business impact. Maintaining optimal index health helps ensure fast, reliable data access, reduced resource consumption, and an overall improvement in the user experience. Consequently, implementing a well-designed index maintenance strategy is a crucial responsibility for any DBA managing a complex SQL Server environment. Ola Hallengren's SQL Server Maintenance Solution The SQL Server Maintenance Solution, developed by Ola Hallengren, is a widely adopted and trusted set of scripts used by database administrators worldwide. This comprehensive solution automates various maintenance tasks, including index optimization, database integrity checks, and statistics updates. Ola's scripts have become the industry standard for proactive database maintenance. The IndexOptimize procedure from the Maintenance Solution provides extensive customization and configuration options to tailor the index maintenance process for specific environments and requirements. Many database administrators rely on these scripts as the foundation for their index management strategy, as they offer a robust and efficient way to keep indexes in an optimal state. You can download the latest SQL Server Maintenance Solution version from Ola Hallengren's website. The scripts are released under the MIT License, allowing users to freely use, modify, and distribute them as needed. Core IndexOptimize Parameters and Their Impact The `IndexOptimize` stored procedure provides extensive customization through numerous parameters. Understanding these is critical for effective implementation: Essential Parameters ParameterDescriptionImpact`@Databases`Target databasesControls scope of operation`@FragmentationLow`Action for low fragmentationTypically NULL (no action)`@FragmentationMedium`Action for medium fragmentationUsually REORGANIZE`@FragmentationHigh`Action for high fragmentationREBUILD or REORGANIZE`@FragmentationLevel1`Low/medium threshold (%)Typically 5-15%`@FragmentationLevel2`Medium/high threshold (%)Typically 30-40%`@PageCountLevel`Minimum index size to processExcludes small indexes`@SortInTempdb`Use tempdb for sortingReduces production database I/O`@MaxDOP`Degree of parallelismControls CPU utilization`@FillFactor`Index fill factorControls free space in pages`@PadIndex`Apply fill factor to non-leaf levelsAffects overall index size`@LOBCompaction`Compact LOB dataReduces storage for LOB columns`@UpdateStatistics`Update statistics after rebuild'ALL', 'COLUMNS', 'INDEX', NULL`@OnlyModifiedStatistics`Only update changed statisticsReduces unnecessary updates`@TimeLimit`Maximum execution time (seconds)Prevents runaway jobs`@Delay`Pause between operations (seconds)Reduces continuous resource pressure`@Indexes`Specific indexes to maintainAllows targeted maintenance`@MinNumberOfPages`Minimum size thresholdAlternative to PageCountLevel`@MaxNumberOfPages`Maximum size thresholdLimits operation to smaller indexes`@LockTimeout`Lock timeout (seconds)Prevents blocking`@LogToTable`Log operations to tableEnables tracking/troubleshootingParameterDescriptionRecommended Setting`@AvailabilityGroups`Target specific AGsLimit scope when needed`@AvailabilityGroupReplicas`Target specific replicas'PRIMARY' to limit AG impact`@AvailabilityGroupDatabases`Target specific databasesFocus on critical databases Availability Group-Specific Parameters ParameterDescriptionRecommended Setting`@AvailabilityGroups`Target specific AGsLimit scope when needed`@AvailabilityGroupReplicas`Target specific replicas'PRIMARY' to limit AG impact`@AvailabilityGroupDatabases`Target specific databasesFocus on critical databases Implementation Strategies by Index Size Large Indexes (>10GB) EXECUTE dbo.IndexOptimize @Databases = 'PRODUCTION_DB', @FragmentationLow = NULL, @FragmentationMedium = 'INDEX_REORGANIZE', @FragmentationHigh = 'INDEX_REORGANIZE,INDEX_REBUILD_ONLINE', @FragmentationLevel1 = 15, @FragmentationLevel2 = 40, @PageCountLevel = 10000, -- Only process substantial indexes @MaxDOP = 4, -- Limit CPU utilization @TimeLimit = 7200, -- 2-hour limit per operation @Delay = '00:00:45', -- 45-second pause between operations @SortInTempdb = 'Y', -- Reduce database file I/O @MaxNumberOfPages = NULL, -- No upper limit @MinNumberOfPages = 10000, @LockTimeout = 300, -- 5-minute lock timeout @LogToTable = 'Y', @Execute = 'Y'; Special considerations: Prefer REORGANIZE for large indexes to minimize transaction log growthUse REBUILD selectively when reorganize is insufficientImplement larger `@Delay`to allow transaction log processingSchedule during low-activity periodsConsider smaller batches using `@Indexes` parameter Medium Indexes (1GB-10GB) EXECUTE dbo.IndexOptimize @Databases = 'PRODUCTION_DB', @FragmentationLow = NULL, @FragmentationMedium = 'INDEX_REORGANIZE', @FragmentationHigh = 'INDEX_REBUILD_ONLINE', @FragmentationLevel1 = 10, @FragmentationLevel2 = 30, @PageCountLevel = 1000, @MaxDOP = 2, @TimeLimit = 3600, -- 1-hour limit @Delay = '00:00:20', -- 20-second pause @SortInTempdb = 'Y', @MinNumberOfPages = 1000, @MaxNumberOfPages = 10000, @LockTimeout = 180, -- 3-minute lock timeout @LogToTable = 'Y', @Execute = 'Y'; Special considerations: Balance between REORGANIZE and REBUILD operationsModerate `@Delay` value to manage resource impactCan run more frequently than large index maintenance Small Indexes (<1GB) EXECUTE dbo.IndexOptimize @Databases = 'PRODUCTION_DB', @FragmentationLow = NULL, @FragmentationMedium = 'INDEX_REORGANIZE', @FragmentationHigh = 'INDEX_REBUILD_ONLINE', @FragmentationLevel1 = 5, @FragmentationLevel2 = 30, @PageCountLevel = 100, @MaxDOP = 0, -- Use server default @TimeLimit = 1800, -- 30-minute limit @Delay = '00:00:05', -- 5-second pause @SortInTempdb = 'Y', @MaxNumberOfPages = 1000, @MinNumberOfPages = 100, @LockTimeout = 60, -- 1-minute lock timeout @LogToTable = 'Y', @Execute = 'Y'; Special considerations: Can be more aggressive with rebuild operations.Minimal `@Delay` needed between operations.Can run during regular business hours with minimal impact. Availability Group-Specific Configurations Environment: Large, mission-critical OLTP database with multiple replicas in an Availability Group (AG) configured for synchronous commit. Maintenance Objectives: Minimize impact on production workload and log shipping.Avoid exhausting storage resources due to log growth.Ensure high availability and minimal downtime. Synchronous AG Environment EXECUTE dbo.IndexOptimize @Databases = 'PRODUCTION_DB', @FragmentationLow = NULL, @FragmentationMedium = 'INDEX_REORGANIZE', @FragmentationHigh = 'INDEX_REORGANIZE', -- Avoid rebuilds in sync AGs @FragmentationLevel1 = 15, @FragmentationLevel2 = 40, @PageCountLevel = 5000, @MaxDOP = 2, @TimeLimit = 3600, @Delay = '00:01:00', -- Longer delay for sync replicas @AvailabilityGroupReplicas = 'PRIMARY', @LockTimeout = 300, @LogToTable = 'Y', @Execute = 'Y'; Synchronous AG considerations: Minimize rebuilds - Transaction logs must be synchronized before the operation completes.Implement longer delays between operations to allow synchronization.Monitor replica lag and suspend jobs if lag exceeds thresholds.Increase log backup frequency during maintenance windows.Split maintenance across multiple days for very large environments. Asynchronous AG Environment Environment: Large, multi-terabyte data warehouse database with asynchronous AG replicas. Maintenance Objectives: Perform comprehensive index and statistics maintenanceMinimize the impact on the reporting workload during the maintenance windowEnsure optimal performance for the upcoming quarter EXECUTE dbo.IndexOptimize @Databases = 'PRODUCTION_DB', @FragmentationLow = NULL, @FragmentationMedium = 'INDEX_REORGANIZE', @FragmentationHigh = 'INDEX_REBUILD_ONLINE', -- Rebuilds more acceptable @FragmentationLevel1 = 10, @FragmentationLevel2 = 30, @PageCountLevel = 2000, @MaxDOP = 4, @TimeLimit = 5400, @Delay = '00:00:30', -- Moderate delay @AvailabilityGroupReplicas = 'PRIMARY', @LockTimeout = 240, @LogToTable = 'Y', @Execute = 'Y'; Asynchronous AG considerations: More liberal with rebuilds - operations don't wait for secondary synchronization.Still monitor send queue to prevent overwhelming secondaries.Consider network bandwidth and adjust `@Delay` accordingly.Implement send queue size alerts during maintenance. Preventing Storage and IOPS Pressure Pre-Maintenance Preparation Expand transaction log files proactively: ALTER DATABASE [YourDatabase] MODIFY FILE (NAME = LogFileName, SIZE = ExpandedSizeInMB); Configure TempDB properly: -- Verify TempDB configuration SELECT name, size/128.0 AS [Size_MB] FROM tempdb.sys.database_files; Implement pre-maintenance checks: -- Create helper procedure to validate environment readiness CREATE PROCEDURE dbo.ValidateMaintenanceReadiness AS BEGIN DECLARE @IssuesFound BIT = 0; -- Check log space IF EXISTS ( SELECT 1 FROM sys.databases d CROSS APPLY sys.dm_db_log_space_usage() l WHERE d.database_id = DB_ID() AND l.log_space_used_percent > 30 ) BEGIN RAISERROR('Log usage exceeds 30%. Backup logs before proceeding.', 16, 1); SET @IssuesFound = 1; END -- Check AG health IF EXISTS ( SELECT 1 FROM sys.dm_hadr_availability_replica_states ars JOIN sys.availability_replicas ar ON ars.replica_id = ar.replica_id WHERE ars.is_local = 0 AND ars.synchronization_health <> 2 -- Not HEALTHY ) BEGIN RAISERROR('Availability Group replicas not in healthy state.', 16, 1); SET @IssuesFound = 1; END RETURN @IssuesFound; END; GO Operational Techniques Implement dynamic index selection based on business impact: -- Create index priority categories CREATE TABLE dbo.IndexMaintenancePriority ( SchemaName NVARCHAR(128), TableName NVARCHAR(128), IndexName NVARCHAR(128), Priority INT, -- 1=High, 2=Medium, 3=Low MaintenanceDay TINYINT -- Day of week (1-7) ); -- Use with dynamic execution DECLARE @IndexList NVARCHAR(MAX); SELECT @IndexList = STRING_AGG(CONCAT(DB_NAME(), '.', SchemaName, '.', TableName, '.', IndexName), ',') FROM dbo.IndexMaintenancePriority WHERE Priority = 1 AND MaintenanceDay = DATEPART(WEEKDAY, GETDATE()); EXEC dbo.IndexOptimize @Databases = 'PRODUCTION_DB', @Indexes = @IndexList, -- other parameters Implement I/O throttling techniques: Use Resource Governor to limit I/O (SQL Server Enterprise).Set lower `@MaxDOP` values during business hours.Implement longer `@Delay` values during peak periods. Database-level I/O tuning: -- Consider trace flag 1117 for uniform file growth DBCC TRACEON(1117, -1); -- Consider trace flag 1118 for reducing SGAM contention DBCC TRACEON(1118, -1); -- For SQL Server 2016+, use proper tempdb configuration ALTER DATABASE [tempdb] MODIFY FILE (NAME = 'tempdev', SIZE = 8GB); Advanced Scheduling Strategies Workload-Aware Batching -- Create helper procedure for smart batching CREATE PROCEDURE dbo.ExecuteIndexMaintenanceBatch @BatchSize INT = 5, @MaxRuntime INT = 7200 -- 2 hours in seconds AS BEGIN DECLARE @StartTime DATETIME = GETDATE(); DECLARE @EndTime DATETIME = DATEADD(SECOND, @MaxRuntime, @StartTime); DECLARE @CurrentTime DATETIME; DECLARE @IndexBatch NVARCHAR(MAX); WHILE (1=1) BEGIN SET @CurrentTime = GETDATE(); IF @CurrentTime > @EndTime BREAK; -- Get next batch of indexes based on priority and fragmentation SELECT TOP (@BatchSize) @IndexBatch = STRING_AGG(CONCAT(DB_NAME(), '.', s.name, '.', t.name, '.', i.name), ',') FROM sys.indexes i JOIN sys.tables t ON i.object_id = t.object_id JOIN sys.schemas s ON t.schema_id = s.schema_id JOIN sys.dm_db_index_physical_stats(DB_ID(), NULL, NULL, NULL, 'LIMITED') ps ON ps.object_id = i.object_id AND ps.index_id = i.index_id WHERE i.type_desc = 'NONCLUSTERED' AND ps.avg_fragmentation_in_percent > 30 AND ps.page_count > 1000 AND NOT EXISTS ( -- Skip indexes we've already processed SELECT 1 FROM dbo.CommandLog WHERE DatabaseName = DB_NAME() AND SchemaName = s.name AND ObjectName = t.name AND IndexName = i.name AND StartTime > DATEADD(DAY, -7, GETDATE()) ) ORDER BY ps.avg_fragmentation_in_percent DESC; IF @IndexBatch IS NULL BREAK; -- No more work to do -- Execute maintenance for this batch EXEC dbo.IndexOptimize @Databases = DB_NAME(), @Indexes = @IndexBatch, @FragmentationLow = NULL, @FragmentationMedium = 'INDEX_REORGANIZE', @FragmentationHigh = 'INDEX_REORGANIZE', @FragmentationLevel1 = 10, @FragmentationLevel2 = 30, @MaxDOP = 2, @TimeLimit = 1800, -- 30 minutes per batch @Delay = '00:00:30', @LogToTable = 'Y', @Execute = 'Y'; -- Pause between batches WAITFOR DELAY '00:01:00'; END END; GO Monitoring Framework -- Create monitoring stored procedure CREATE PROCEDURE dbo.MonitorIndexMaintenance AS BEGIN -- Check transaction log usage SELECT DB_NAME(database_id) AS DatabaseName, log_space_in_use_percentage FROM sys.dm_db_log_space_usage WHERE log_space_in_use_percentage > 50; -- Check AG send queue size SELECT ar.replica_server_name, drs.database_name, drs.log_send_queue_size, drs.log_send_rate, drs.redo_queue_size, drs.redo_rate FROM sys.dm_hadr_database_replica_states drs JOIN sys.availability_replicas ar ON drs.replica_id = ar.replica_id WHERE drs.log_send_queue_size > 10000 OR drs.redo_queue_size > 10000; -- Check ongoing index operations SELECT r.session_id, r.command, r.status, r.wait_type, r.wait_time, OBJECT_NAME(p.object_id) AS ObjectName, p.index_id, i.name AS IndexName FROM sys.dm_exec_requests r CROSS APPLY sys.dm_exec_sql_text(r.sql_handle) t LEFT JOIN sys.partitions p ON p.hobt_id = r.statement_id LEFT JOIN sys.indexes i ON i.object_id = p.object_id AND i.index_id = p.index_id WHERE t.text LIKE '%INDEX_REBUILD%' OR t.text LIKE '%INDEX_REORGANIZE%'; END; GO Best Practices Summary For synchronous AG environments: Prioritize REORGANIZE over REBUILD, especially for large indexes.Implement longer delays between operations (45-90 seconds). Schedule maintenance during the least active periods.Consider partitioning very large tables for incremental maintenance. For asynchronous AG environments: More liberal use of REBUILD for critical indexes. Implement moderate delays (15-45 seconds). Monitor send queue and redo queue sizes closely. General IOPS reduction techniques: Leverage `@SortInTempdb = 'Y'` to spread I/O load. Use `@MaxDOP` to control parallelism (lower values reduce I/O).Implement `@Delay` parameters appropriate to your environment. Use `@TimeLimit` to prevent runaway operations. Storage pressure mitigation: Pre-allocate transaction log space before maintenance.Increase log backup frequency during maintenance (every 5-15 minutes). Use Resource Governor to limit I/O impact. Implement batched approaches with appropriate pauses. Comprehensive maintenance approach: Different strategies for different index sizes. Business-hour vs. off-hour configurations. Prioritization based on business impact.Regular verification of fragmentation levels post-maintenance. By implementing these guidelines and adapting the provided scripts to your specific environment, you can maintain optimal SQL Server index performance while minimizing production impact, even in complex Availability Group configurations.

By arvind toorpu CORE

Along with the rise of Kubernetes, there is another shift that is happening under the hood - the rise of serverless architecture, which is quietly rewriting the way we deploy and scale applications, with Java taking a lead. Java, which is usually associated with legacy code and monolithic enterprise applications, has been slowly but steadily evolving into a microservices architecture and is now evolving into a leaner, serverless-ready world. With the availability of tools like Knative and frameworks like Quarkus, Java has been transforming from a heavyweight language into a zero-management, Kubernetes-ready ready. In this article, we will reflect on this promising transformation in Java and where it can take us in 2025 and beyond. The Promise of Serverless Kubernetes Architecture Kubernetes is really good at handling containers; however, the management of the underlying infrastructure, like nodes, scaling, and patching, is still difficult. However, serverless Kubernetes frameworks like Knative, KEDA, and OpenFaaS remove this difficulty; they let applications scale to zero and back up without the need for any human intervention. This approach not only enhances developer productivity but also helps with cost efficiency and is in sync with the needs for a modern event-driven system. Java has also been slowly catching up on this. Historically, Java has been associated with resource-intensive applications, where there were challenges with startup speed and being a memory-intensive language. However, with recent advancements, Java is becoming a viable option for serverless deployments. Java's Technical Evolution Java's adaptation to a serverless environment is primarily driven by optimized frameworks and runtime enhancements like, 1. Quarkus Quarkus was introduced by Red Hat. It is designed for Kubernetes native deployment and integrates well with GraalVM. This enables native compilation, which reduces startup times and lowers memory usage, making applications built with Java suitable for serverless scenarios. This framework also simplifies the developer experience by offering a unified approach to both imperative and reactive coding styles. 2. Spring Boot Enhancements Recent versions of Spring Boot have incorporated several features that are aimed at improving performance in cloud native environments. This includes features like support for reactive programming models, integration with Kubernetes, among others. These updates allow developers to seamlessly bridge traditional Java applications with modern, distributed systems. 3. Project Loom Project Loom is an initiative to introduce lightweight, user-mode threads to the Java Virtual Machine (JVM), which aims to enhance concurrency handling. It does so by enabling the JVM to manage numerous virtual threads efficiently, which improves the scalability of Java applications in I/O-bound operations. This can be beneficial for serverless workloads. This innovation promises to make Java a go-to choice for highly concurrent workloads without the complexity of traditional threading models. Practical Use Cases and Applications Java's applicability in a serverless Kubernetes environment can be understood with different use cases as discussed below: 1. Event-Driven Architecture Java frameworks like Quarkus can be used to build event-driven applications that respond to various triggers, such as HTTP requests or messaging queues. This capability allows Java to power dynamic systems where responsiveness to unpredictable events is critical. 2. Microservices Java has a rich ecosystem that supports the development of microservices, which can be deployed in serverless environments and benefit from automatic scaling and reduced operational overhead. Developers can now build modular, independent services that integrate smoothly with existing enterprise solutions. 3. Batch Processing and AI Inference Java can be employed in batch processing for large-scale batch jobs and AI inference tasks, taking advantage of serverless platforms that auto-scale with workload demands and minimize idle resource costs. Its robust libraries make it particularly effective for processing intricate datasets or deploying machine learning models in real time. Challenges and Limitations Although Java is making significant advancements, there are several challenges, 1. Cold Start Latency Even though there are optimizations happening, Java applications can still experience higher cold start latencies compared to other languages like NodeJS, which can be an issue in applications where latency is monitored. This issue often stems from the inherent overhead of initializing the JVM in transient environments. 2. Build Complexity Native compilation with tools like GraalVM can introduce additional build complexities. Also, if there is a misconfiguration, it can lead to runtime issues or an increase in the size of a binary file. Developers may also find debugging these native images trickier than traditional Java setups. 3. Learning Curve Frameworks like Knative can be difficult to learn initially and can have a learning curve, particularly for advanced features like eventing, which may require significant investment in understanding and its implementation. This complexity can slow adoption in teams accustomed to simpler deployment models. Competitive Edge Java has an established ecosystem that offers a wealth of libraries and frameworks, such as JPA for data persistence and Spring Security for authentication and authorization. This maturity in the framework allows developers to leverage existing tools and practices, which facilitates an easy transition to serverless architecture. Java's use across various industries would facilitate easy adaptability of existing skills with the new paradigm due to the vast community of developers who use Java. Additionally, Java’s long-standing reputation for reliability ensures it remains a trusted choice for mission-critical applications in serverless contexts. Future Implications With Java continuously evolving through projects like Loom and Quarkus, its role and usage for applications built for serverless environments are expected to grow. The combination of various factors, including scaling up from zero, simplicity in operations, and agility in deployments, positions Java as a strong contender for future cloud-native applications. As serverless adoption expands, Java’s ability to integrate with emerging tools and platforms will keep it relevant in an ever-evolving tech landscape. Its ongoing evolution suggests it could redefine how enterprises approach scalable, cost-effective solutions well into the future. Conclusion The transformation of Java in the context of serverless Kubernetes not only reflects a strategic evolution, but it also leverages modern frameworks and runtime enhancements to meet the demands of cloud-native development. Though there has been some progress, some challenges remain. However, with the trajectory Java is taking, it is clear that it is not only adapting to new demands but also thriving in this paradigm. With the strong march, Java has positioned itself such that developers and architects can confidently consider Java as a robust and scalable foundation for serverless applications.